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PREFACE 


This  report  is  one  of  a aeries  of  guidebooks  intended  to  help  Program 
Office  personnel  in  software  acquisition  management.  The  contents  of  the 
guidebooks  will  be  revised  periodically  to  reflect  changes  in  software 
acquisition  policies  & practices,  and  feedback  from  users. 

This  guidebook  has  been  prepared  under  the  direction  of  the  Electronic 
Systems  Division  (ESD),  Air  Force  Systems  Command  (AFSC),  Computer  Systems 
Engineering  Directorate  (MCI).  Contributions  were  made  by  Captain  W.  J.  White 
(MCI)  (Project  Officer). 

The  Software  Acquisition  Management  Guidebook  series  is  currently  planned 
to  cover  the  following  topics.  (National  Technical  Information  Service 
accession  numbers  for  those  published  to  date  are  in  parentheses). 

1.  Project  Guide  to  Content  Requirement  and  Audience  Needs  (AD-A019124) 

2.  Regulations,  Specifications  & Standards  (AD-AO 16401) 

3.  Contracting  for  Software  Acquisition  (AD-A020444) 

4.  Monitoring  and  Reporting  Software  Development  Statue  (AD-Av,'**  ^ 

5.  Statement  of  Work  Preparation 

6.  Reviews  and  Audits 

» 

7.  Configuration  Management 

b.  Requirements  Specification 

9.  Software  Documentation  Requirements  (AD-A027051) 

10.  Verification 

11.  Validation  and  Certification 

12.  Overview  of  the  Series 

13-  Computer  Program  Maintenance 

14.  Software  Quality  Assurance 

15.  Software  Cost  Estimating  and  Measuring 

16.  Software  Development  and  Maintenance  Facilities 

17.  Life  Cycle  Events 
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1 . INTRODUCTION 


This  guidebook  explains  the  Acquisition  Life  Cycle*  for  Major  Defense 
Systems  (defined  in  Section  1.1),  the  Computer  Program  Life  Cycle,  and  their 
relationships.  Some  knowledge  of  these  topics  is  essential  to  understanding 
the  roles  of  the  Government  organizations  and  contractors  in  the  acquisition 
of  Electronic  Systems  that  Include  software.  Electronic  Systems  are  one  of 
seven  types  of  system  identified  in  MIL-STD-881A,  Work  Breakdown  Structures 
for  Defense  Material  Items.  A substantial  number  of  ESD-managed  systems  are 
Electronic  Systems.  This  material  is  presented  partly  for  the  benefit  of 
those  who  may  be  assigned  to  prepare  or  review  software-related  acquisition 
documents  without  formal  training  or  extensive  experience  in  Air  Force  system 
acquisition  programs  involving  software.  The  guidebook  also  identifies  many 
tasks  that  should  be  considered  for  possible  incorporation  in  Statements  of 
Work  (sows)**,  notes  appropriate  products  of  these  tasks,  and  discusses  the 
Regulations,  Specifications  and  Standards  that  prescribe  and  define  them. 

The  guidebook's  treatment  of  these  topics  is  minimal.  It  consists  mainly 
of  short  summaries,  plus  references  to  Regulations,  Specifications,  Standards, 
and  other  sources  that  provide  more  definitive  information.  These  references 
should  be  reviewed  by  those  who  need  a more  thorough  grasp  of  the  topics 
addressed . 

1.1  Purpose 

The  guidebook  has  been  prepared  for  use  by  Air  Force  Program  Office  (PO) 
personnel  in  general  and  a person  termed  the  Software  Director  in  particular. 
The  Software  Director  is  the  military  officer  or  civilian  within  the  Program 
Office  who  assists  the  Program  Manager  (PM)  in  planning  and  managing  software 
development  activities.  As  such,  the  Software  Director  is  one  member  of  an 
Air  Force  program  management  team  that  includes  technical,  procurement,  legal, 
data  management,  configuration  management,  and  other  specialists  whose 
combined  efforts  are  necessary  for  the  successful  completion  of  an  acquisition 
program.  Different  individuals  (e.g.,  the  Engineering  Division  director)  may 
perform  the  Software  Director's  functions  in  different  Program  Offices,  or 
these  functions  may  be  split  among  different  persons.  However,  with 
appropriate  allowance  for  such  variations  in  organization,  this  guidebook's 
contents  apply  unchanged. 

Unlike  a directive,  this  guidebook  does  not  prescribe  what  must  be  done. 
Instead,  it  Identifies  issues  and  pitfalls;  references  relevant  sections  of 
appropriate  Regulations,  Specifications  and  Standards;  and  suggests 
alternative  approaches.  Any  questions  that  may  arise  over  the  feasibility  or 
legality  of  suggestions  made  herein  should  be  referred  for  decision  to  the 
Program  Manager  or  to  the  appropriate  Procuring  Contracting  Officer  (PCO). 


* The  guidebook  capitalizes  specialized  terminology.  See  Section  1.3. 

**  See  the  Software  Acauialtlon  Management  Guidebook;  Statement  of  Work 

Preparation  (abbreviated  SOWG). 
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1 .2  Scope 


The  guidebook  explains  the  chief  activities,  events,  products,  and 
software-related  effort  that  normally  occur  during  the  life  cycles  of  major 
Electronic  Systems  acquired  within  the  framework  of  the  800-series  of  Air 
Force  regulations  and  manuals.  The  800-series  normally  governs  acquisition  of 
computers  and  software  which  are  embedded  in  a weapons  or  command  and  control 
system.  Some  of  this  software  (e.g..  Application  Programs)  may  be  built 
expressly  for  the  weapons  or  command  and  control  system.  Some  (e.g.,  certain 
Operational  Executives)  may  be  modified  versions  of  off-the-shelf  software.  A 
third  subset  (e.g..  Compilers,  Assemblers)  may  consist  of  unaltered  off-the- 
shelf  software.  The  800-series  covers  the  research,  design,  development, 
engineering,  testing,  and  production  of  tactical  & strategic  systems  for  the 
operational  inventory.  In  contrast,  the  acquisition  of  off-the-shelf, 
commercially  marketed  data  processing  equipment  and  its  associated  support 
("non-functional")  software  for  business-like  applications  (e.g.,  payrolls, 
logistics,  personnel  records,  management  reporting)  is  normally  governed  by 
the  300-series  of  Air  Force  regulations  and  manuals.  ESD-TR-75-9 1 , Software 
Acquisition  Management  Guidebook:  Regulations.  Specifications  and  Standards. 
Chapter  2,  further  compares  the  300-series  and  the  800-series.  This  Life 
Cycle  Events  guidebook  does  not  address  acquisitions  managed  in  accordance 
with  the  300-series,  although  some  of  its  principles  may  apply  there  and 
elsewhere. 

1 . 3 Conventions 

The  Regulations,  Specifications  and  Standards  on  which  this  guidebook  is 
largely  based  use  many  terms  drawn  from  ordinary  English  in  special  ways. 

These  directives  define  acronyms  for  some  of  these  terms  but  not  for  others. 
Where  acronyms  are  used,  they  help  make  clear  the  special  meanings  Intended, 
but  where  no  acronym  is  used  confusion  may  arise.  To  minimize  this  problem  in 
the  guidebook  terms  used  in  special  ways  are  capitalized.  These  special  terms 
are  usually  defined  in  the  guidebook  section  where  they  first  occur,  or  in 
references  cited  there.  The  guidebook  uses  acronyms  in  common  parlance,  and 
certain  others  for  brevity.  Each  is  defined  where  first  used,  apd  repeated  in 
the  List  of  Abbreviations. 

Readers  can  distinguish  the  direction,  advice,  and  other  options 
interspersed  in  the  guidebook  by  noting  the  following  convention' . To 
designate  mandatory  action  (e.g.,  action  prescribed  by  applicable  Regulations, 
Specifications  and  Standards)  the  guidebook  employs  "must"  or  "shall".  In 
contrast,  "should"  or  "it  is  recommended  that",  identify  action  recommended  by 
the  author,  while  "may"  and  "might"  connote  other  optional  actions. 

1.4  Plan 

Section  2 introduces  the  Major  Defense  System  Acquisition  Life  Cycle. 
Sections  3-5,  respectively,  summarize  the  chief  activities,  events,  and 
products  of  its  Conceptual  Phase,  its  Validation  Phase,  and  its  Full-Scale 
Development  Phase.  Section  6 deals  with  its  Production  and  Deployment  Phases. 
Section  7 discusses  the  application  of  Major  Defense  System  Acquisition  Life 
Cycle  events  to  Less-Than-Major  Systems.  Section  8 explains  the  Computer 
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Program  Life  Cycle  and  its  relationship  to  the  Acquisition  Life  Cycle. 
Appendix  A discusses  the  Specifications,  because  of  their  special  Importance 
in  system  definition  and  acquisition.  The  guidebook  also  includes  a List  of 
Abbreviations  and  a list  of  pertinent  references. 

The  guidebook's  organization  anticipates  its  use  in  two  main  ways: 

‘ ' I 

a.  as  a tutorial  for  persons  relatively  inexperienced  in  the 
acquisition  of  large  systems  that  include  software; 

b.  as  a summary  of  material  relevant  to  software  acquisition  for  those 
otherwise  quite  familiar  with  the  acquisition  of  large  systems. 


2. 


THL  ACQUISITION  LIFE  CYCLE 


MIL-STD-881A  (paragraph  3.14)  del'ines  Acquisition  as 

"the  aggregation  of  efforts  to  develop,  produce  and  provide  a weapon 
system  to  the  user.  It  commences  in  the  conceptual  phase  and  is 
completed  at  such  time  as  the  last  production  unit  is  provided  to  the 
user . " 

Air  Force  Regulation  (AFR)  800-2,  Program  Management  (Attachment  1,  paragraph 
4),  defines  the  Acquisition  Life  Cycle  for  Major  Defense  Systems  as  normally 
comprising  five  sequential  phases:  Conceptual,  Validation,  Full-Scale 

Development,  Production  and  Deployment.  Major  Defense  System  (i.e.,  "itajor 
program")  status  is  assigned  by  the  Secretary  of  Defense  or  his  Deputy  to  a 
system  whose  acquisition  is  planned,  based  on  estimated  Research,  Development, 
Test,  and  Evaluation  cost  greater  than  $50  million,  estimated  production  cost 
greater  than  $200  mi-Llion,  national  urgency,  or  other  important 
considerations.  Defense  Systems  Acquisition  Review  Council  (DSARC)*  review 
normally  follows  each  of  the  first  three  phases,  after  each  of  which  a 
favorable  decision  by  the  Secretary  of  Defense  is  required  for  the  acquisition 
to  proceed  into  the  next  phase.  AFR  800-2  terms  these  three  decisions  the 
Program  Decision,  the  Ratification  Decision  and  the  Review  Decision,** 
respectively. 

DODI  5000.2  (paragraph  IV. B)  defines  the  same  decision  points  slightly 
differently.  It  states  that  these  are  normal,  but  also  provides  for  different 
or  additional  major  decision  points  established  jointly  by  the  Military 
Services  and  the  Office  of  the  Secretary  of  Defense  (OSD)  if  they  deem  it 
worthwhile.  Each  of  these  decision  points  permits  the  Secretary  of  Defense  to 
redirect  a major  program  in  trouble,  or  to  cancel  it,  without  total  loss  of 
planned  investment. 

While  the  agendas  of  the  different  DSARC  reviews  differ  significantly 
because  the  system  under  review  changes  during  development,  all  DSARC  reviews 
have  certain  common  objectives.  These  include  assuring  continuing  operational 
need,  adequate  system  performance,  acceptable  cost,  and  favorable  cost 
effectiveness  relative  to  other  alternatives.  Naturally,  the  anticipated 
agenda  of  each  DSARC  review  strongly  influences  the  work  done  in  the 
Acquisition  Life  Cycle  phase  that  culminates  in  that  review. 


* Department  of  Defense  Instruction  (DODI)  5000.2,  The  Decision 

Coordinating  Paper  (DCP)  and  the  Defense  Systems  Acquisition  Review 
Council  (DSARC).  and  Department  of  Defense  Directive  (DODD)  5000.26, 
Defense  Systems  Acquisition  Review  Council  (DSARC)  define  DSARC 
composition,  responsibilities  and  operating  procedures.  These  directives 
are  included  as  Attachments  445.  respectively,  to  AFR  800-2. 

**  AFSCP  800-3.  A Guide  to  Program  Management,  terms  this  the  Production 
Decision^ 
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A Decision  Coordinating  Paper  (DCP)  must  be  prepared  to  support  each 
normal  DSARC  review.  These  DCPs  are  termed  DCP  I,  DCP  II,  and  DCP  III, 
respectively.  Limited  to  20  pages,  each  DCP  is  required*  to  record  the 
essential  information  about  the  system  and  its  status  (e.g.,  need,  threat, 
concept,  milestones,  unresolved  issues),  and  eventually  the  Secretary  of 
Defense's  decision.  The  latest  DCP  is  also  expected**  to  be  reviewed  annually 
and  revised  as  necessary  to  reflect  significant  program  changes;  e.g.,  cost 
estimates.  Thus,  review  of  a system's  latest  DCP  version,  if  available,  is 
strongly  recommended  as  important  background  information  in  compact  form. 

Assuming  the  normal  three  decision  points,  the  objectives,  initial 
conditions,  major  activities,  and  major  products  of  the  Acquisition  Life  Cycle 
phases  are  outlined  in  Sections  3-6.  However,  note  that  under  appropriate 
circumstances  a program  may  skip  a phase#  (e.g.,  the  Production  Phase  in 
acquisition  of  a one-of-a-kind  Command,  Control  and  Communications  system). 

Tables  1,  2,  and  3,  respectively,  summarize  the  major  types  of  activity, 
other  events,  products,  and  software-related  effort  that  occur  in  each  of  the 
Conceptual  Phase,  the  Validation  Phase,  and  the  Full-Scale  Development  Phase 
of  the  Acquisition  Life  Cycle.  Some  of  these  products  are  contractor-prepared 
documents  whose  content  and  format  are  prescribed  by  Government  Data  Item 
Descriptions  (DIDs)  (see  SOWG,  Section  C2.7).  In  each  of  Tables  1-3,  the  name 
or  acronym  of  each  such  document  is  followed  in  brackets  by  its  DID's 
identifier.  Further  information  about  most  of  the  document  types  mentioned  in 
this  guidebook  may  be  found  in  ESD-TR-76-159 , An  Air  Force  Guide  to  Software 
Documentation  Requirements. 

These  tables  also  indicate  typical  roles  of  Government  participants  and 
types  of  contractor  support  that  may  be  appropriate.  The  Government  roles  can 
vary  considerably  from  program  to  program.  For  each  Air  Force-managed  Major 
Defense  System,  a Program  Management  Directive  (PMD)  specifies  these  roles. 
Some  of  the  most  important  table  entries  are  further  explained  in  Sections  3- 
5.  A type  of  activity  or  other  event  is  mentioned  in  one  of  Tables  1-3  if  it 
satisfies  any  of  the  following  criteria. 

a.  It  entails  software-related  effort  properly  done  only  by  Government 
(including  Federal  Contract  Research  Center  (FCRC))  personnel;  for 
example,  preparation  of  independent  cost  estimates  is  such  an 
activity. 

b.  It  involves  software-related  work  appropriate  for  a contractor## 
(e.g.,  application  computer  program  development). 


* DODI  5000.2,  paragraph  IV. A. 2. 

**  DODI  5000.2,  Enclosure  1,  paragraph  I.G. 

# AFR  800-2,  Attachment  1,  paragraph  1. 

##  ESD-TR-75-365,  Ah  Air  Force  Guide  to  Contracting  for  Software 

AcQulsitlon . provides  an  overview  of  what  such  contracting  Involves. 
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c.  It  directs  policies,  actions,  organizational  relationships  or  other 
constraints  on  the  system  (e.g.,  use  of  a particular  computer  and 
executive  software)  that  may  affect  acquisition. 

Analogous  tables  for  the  Production  Phase  and  the  Deployment  Phase  are 
not  provided,  because  these  phases'  simpler  software-related  activities  are 
readily  summarized  in  text  (see  Section  6).  The  acquisition  of  systems  that 
do  not  qualify  as  major  programs  is  touched  on  in  Section  7. 

The  activities,  events,  products  and  Government  roles  mentioned  in  Tables 
1-3  and  in  Sections  3-6  are  based  mainly  on  interpretation  of  AFSCP  800-3,  on 
APR  800-14,  Vol.  II,  Acquisition  and  Support  Procedures  for  Computer  Resources 
in  Systems,  on  AFR  800-2,  and  on  DODI  5000.2.  The  tabular  material  is  grouped 
somewhat  differently  than  in  its  sources.  The  corresponding  types  of 
software-re’ ated  effort  have  been  identified  partly  on  the  basis  of  the 
author's  ac.juisltlon  program  experience.  Each  of  these  tasks  is  either 
necessary  to  develop  a required  product;  or  else  is  usually  essential  to 
accurate  forecasting,  to  sound  design  and  planning,  or  to  good  management  of 
software  development.  The  tables  also  suggest  the  type(s)  of  contractor 
support  (if  any)  that  may  be  appropriate  to  each  such  task. 

Note  that  contractor  support  is  never  mandatory.  Given  enough  expert 
manpower.  Government  (including  FCRC)  personnel  may  do  almost  any  software- 
related  task  as  well  as  a contractor.  Some  types  of  task,  including  Technical 
Performance  Prediction  (e.g.,  computer  simulation  and  analysis  of  system 
response  times),  may  be  done  better  by  technically  qualified  Government 
personnel  than  by  contractor  personnel,  because  typically  greater  insight, 
faster  response  to  change,  and  better  control  are  then  possible. 


10 


3. 


CONCEPTUAL  PHASE 


3- 1 Ob lectlves 

The  Conceptual  Phase  has  two  primary  goals.  The  first  is  to  explore, 
formulate  and  evaluate  possible  requirements  for  a new  or  significantly 
improved  Major  Defense  System.  Second,  if  the  need  appears  great  enough,  the 
Conceptual  Phase  work  should  devise,  for  DSARC  and  Secretary  of  Defense 
review,  an  optimum,  affordable,  and  coat  effective  preferred  approach  to  the 
system's  development,  production,  and  deployment.  In  support  of  this  goal, 
considerable  preliminary  design  and  analysis  of  software  may  be  appropriate. 
Except  for  development  of  demonstration,  prototype,  and  simulation  software 
such  Conceptual  Phase  software  design  and  analysis  should  normally  be  limited 
in  level  and  scope  to  whatever  is  necessary  to  establish  technical  feasibility 
and  credible  estimates  of  costs  and  development  times.  This  level  will  vary 
from  function  to  function.  Design  and  analysis  should  normally  be  most 
detailed  where  technical  risk  is  greatest. 

3* ••2  Initiating  Events 

Refer  to  Table  1,  Sets  A-F.  An  Air  Force  system's  Conceptual  Phase  may 
be  said  to  have  started  whenever  the  Department  of  Defense  (DoD),  the  Air 
Staff,  or  a major  Air  Force  command  directs  studies  that  reveal  serious 
deficiencies  in  some  aspect  of  our  national  defense  posture  and  which  suggest 
a promising  approach  to  their  correction.  One  commoii  type  of  Conceptual  Phase 
initiating  event  is  a major  command's  submission  to  the  Air  Staff  of  a 
Required  Operational  Capability  (ROC)»  (Table  1,  Set  C).  A ROC  describes 
deficiencies  in  a command's  systems  that  prevent  it  from  fully  meeting  its 
responsibilities;  the  ROC  may  also  suggest  new  or  improved  corrective 
capabilities.  Another  type  of  initiating  event  could  be  the  formation  of  a 
special  Mission  Analysis  Steering  Group,  (Table  1,  Set  B),  chaired  by  a 
specific  Air  Force  operational  command,  to  explore  alleged  deficiencies  in  a 
system  or  mission  area.  Conceptual  Phase  activity  could  also  begin  informally 
as  a result  of  needs  revealed  by  routine  planning  studies.  If  its  review  of 
the  ROC  Evaluation,  Mission  Analysis  Steering  Group  Report,  or  other  planning 
studies  is  favorable.  Headquarters  USAF  will  issue  an  initial  PMD.  This  PMD 
(Table  1,  Set  E)  constitutes  the  authority  to  establish  the  PO  Cadre  (Table  1, 
Set  F)  and  to  begin  major  expenditure  on  Conceptual  Phase  effort.  The  PMD  may 
also  direct  specific  studies  or  development  considered  desirable. •• 

3-3  Other  Activities  and  Related  Products 

Regardless  of  how  it  begins,  the  Conceptual  Phase  will  typically  include 
the  activities,  and  yield  the  related  products,  outlined  in  Table  1,  Sets  G-U. 
Considerable  variation  in  these  activities  and  products  will  occur  among  major 
programs,  because  of  differences  in  formal  direction  (e.g.,  in  the  terms  of 
PMDs)  and  in  local  management  decisions  (e.g.,  by  the  Program  Manager). 


* AFR  57-1,  Required  Operational  Capabilities  (ROCt,.  . 

••  AFR  70-15,  Source  Selection  Policy  and  Procedures,  paragraph  2- la. 
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$t  Tl>«  CovprnMDt  rolts  and  types  of  contractor  support  indicated 
apply  to  the  entire  set  (e.g.,  Set  A,  Set  C),  or  to  the  parts 
of  the  set  (e.g.,  A.I.),  with  the  description  of  which  they  line  up 


However,  the  requirements  for  preparation  of  DCP  I and  for  DSARC  review,  as 
specified  in  DODI  5000.2,  tend  to  standardize  Conceptual  Phase  activity 
somewhat  for  all  Major  Defense  Systems.  Besides  the  initial  draft  of  DCP  I, 
the  Functional  Baseline  (defined  in  AFSCP  800-3,  paragraph  2-21)  and  related 
management  planning  documentation  (see  Table  1,  Sets  T and  U)  are  the  chief 
Conceptual  Phase  products.  Most  of  the  other  activities  mentioned  in  Table  1 
develop  preliminary  versions  of  similar  products,  and  illustrate  the  Iterative 
nature  of  much  Conceptual  Phase  work. 

The  Functional  Baseline  includes  the  initial  version  of  the  System 
Specification.*  This  Initial  System  Specification  should  state  the  system's 
overall  functional,  performance,  interface,  design,  and  testing  requirements. 
In  addition,  it  should  incorporate  the  system's  first-level  design,  by 
identifying  major  parts  of  the  system  (termed  Functional  Areas),  by  defining 
the  interfaces  among  them,  and  by  allocating  among  them  the  system's 
requirements.  If  dividing  the  system  into  System  Segments  (see  Section  *».3.3) 
is  under  consideration,  the  Initial  System  Specification  may  also  identify 
these  System  Segments  and  the  Functional  Areas  belonging  to  each.  Conceptual 
Phase  first-level  design  is  preliminary.  It  is  subject  to  change  as  a result 
of  Validation  Phase  system  definition  and  system  design  validation  activities 
(see  Section  4.3). 

Note  that  several  sets  of  Conceptual  Phase  activities  mentioned  in  Table 
1 include  possible  software-related  work  that  might  be  contracted  for, 
entailing  preparation  of  one  or  more  Conceptual  Phase  RFPs.  This  work 
includes  preparation  of  Validation  Phase  Feasibility  Demonstration  software 
(Table  1,  Set  G);  preliminary  system  design  studies  (Set  I);  system  and 
subsystem  simulation  development,  execution  & modification  (Set  J);  and 
drafting  the  Initial  System  Specification  (Set  T).  Such  work  will  tend  to 
educate  participating  contractor  personnel  in  the  system's  requirements.  The 
Government  may  later  benefit  from  this  expertise  if  the  participating 
Conceptual  Phase  contractors  win  related  Validation  Phase  or  Full-Scale 
Development  Phase  contracts.  However,  to  avoid  grounds  for  possible  claims  of 
bias  by  unsuccessful  Offerers,  competitive  Validation  Phase  and  Full-Scale 
Development  Phase  Requests  for  Proposal  (RFPs)  should  be  structured  to  give  a 
fair  chance  to  Offerers**  without  previous  involvement  in  the  system.  For 
example,  an  RFP  should  allow  a reasonable  amount  of  time  for  competent 
Offerers  to  digest  its  system-specific  material  and  to  prepare  sound 
proposals.  < 

During  the  Conceptual  Phase,  Government  personnel  must  prepare  a 
Preliminary  Project  Summary  Work  Breakdown  Structure  (WBS)  (Table  1,  Set  N), 


AFN  800-14,  Vol.  II,  paragraph  2-3.  The  System  Specification  is  defined 
in  MIL-STD-490,  Specification  Practices,  paragraphs  3* 1.3.1  and  10;  and 
in  MIL-STD-483(USAF),  Configuration  Management  Practlcea  for  Systems. 
Equipment.  Munitions,  and  Computer  Programs,  paragraph  30. 

Companies  that  submit  proposals  are  termed  Offerers. 
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alternate  Program  Breakdown  Structures  (PBSa)(Table  1,  Set  P),  a Source 
Selection  Plan,*  a Procurement  Plan,**  and  any  Validation  Phase  RFP(s)  (Table 
1,  Set  U).  The  Software  Director  should  prepare  the  software-related  portions 
of  these  documents,  and  review  the  other  portions.  Other  more  specific  plans 
may  be  obtained  from  Validation  Phase  (or  Full  Scale  Development  Phase) 
contractors  by  requiring  them  in  RFPs.  Noteworthy  examples  include  a Computer 
Program  Development  Plan  (CPDF)  and  a System  Engineering  Management  Plan 
(SEMP).  These  are  discussed  in  Sections  ^1.4.5  and  4.4.6. 

Note  that  appointment  of  the  Program  Manager  and  PO  Cadre  formation 
(Table  1,  Set  F)  occur  only  after  several  significant  Conceptual  Phase 
activities  have  begun.  These  early  Conceptual  Phase  activities  are  managed  by 
planning  staffs  at  Intermediate  Commands  (e.g.,  ESD/XR). 

3-4  Terminating  Events 

Refer  to  Table  1,  Set  V.  The  Conceptual  Phase  has  no  prescribed  time 
limit.  Before  DSARC  review  of  the  draft  DCP  begins,  the  program  can  be 
terminated  with  the  approval  of  the  highest  command  level  which  authorized  it. 
DSARC  review  follows  a formal  request  by  the  Secretary  of  Air  Force  (SAF). 

Once  DSARC  review  begins,  the  Conceptual  Phase  will  normally  end  with  the 
Secretary  of  Defense's  Program  Decision  to  proceed  into  the  Validation  Phase 
(with  or  without  specific  redirection),  or  to  end  the  program. 


* AFR  70-15,  paragraph  2-2. 

**  Air  Force  Armed  Services  Procurement  Regulation  (ASPR)  Supplement  1- 
2100.50,  and  ESD-TR-75-365 » paragraph  2.3*3* 
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VALIDATION  PHASE 


4. 1 Qb  iectives 

The  first  main  Validation  Phase  goal  is  to  assess  the  Major  Defense 
System's  preferred  design  approach,  selected  as  a result  of  Conceptual  Phase 
activity  (see  Section  3*1)i  against  the  system's  requirements  (e.g.,  as  stated 
in  its  Initial  System  Specification).  If  this  approach  proves  unsatisfactory, 
a reasonable  effort  should  be  made  to  rectify  it,  or  to  develop  and  verify  a 
better  one.  If  and  when  a sound  system  design  approach  is  achieved,  the 
second  main  Validation  Phase  goal  is  to  provide  sound  technical,  contractual, 
economic,  and  organizational  bases  for  the  system's  Full-Scale  Development. 

4.2  Initiating  Events 

Refer  to  Table  2,  Set  A?  The  Validation  Phase  begins  with  a favorable 
Program  Decision  (and  possible  supplementary  direction)  by  the  Secretary  of 
Defense.  Supplementary  guidance  from  Headquarters  USAF  and  from  AFSC  follows. 
This  direction  is  consolidated  in  a revised  PMD*  and  AFSC  Form  56. 

4.3  System  Definition  and  Validation 

Refer  to  Table  2,  Sets  B 4 E.  As  defined  in  the  Regulations, 
Specifications  and  Standards,  Valioation  Phase  technical  activities  consist 
mainly  of  work  to  demonstrate  the  feasibility  of  doubtful  components  and 
subsystems,  to  refine  the  selected  system  design  and  interface  definitions, 
and  to  improve  related  estimates  of  performance,  cost  and  schedule.  All  can 
be  considered  •'isk-reduction  mear^res. 

In  addition,  it  may  be  advisable  to  conduct,  during  the  Validation  Phase, 
a design  competition  open  to  industry,  intended  to  develop  if  possible,  and  to 
verify,  a better  design  than  the  preferred  Conceptual  Phase  system  design 
alternative.  Such  a design  competition  is  especially  desirable  if  the 
Conceptual  Phase  design  effort  was  hasty  or  narrowly  based.  Besides 
soliciting  better  design  concepts  from  new  sources,  a design  competition  can 
prevent  or  counter  charges  of  unfair  competitive  practices.  As  an  incentive, 
the  design  competition  should  be  defined  to  accord  the  winner(s)  a substantial 
Full-Scale  Development  Phase  role. 

The  Validation  Phase  is  intended  both  to  reduce  risk  significantly  and  to 
allow  negotiation  of  clear  contracts  (or  analogous  clear  agreements  among 
Government  participants)  for  the  subsequent  acquisition  phases.  Thus, 
unambiguous  specification  of  feasible  and  testable  requirements  during  the 
Validation  Phase  is  most  important.  During  Full-Scale  Development, 
significant  disputes  between  the  Government  and  a contractor  or  between  the 
Implementing  Command  (e.g.;  AFSC,  represented  by  the  Program  Office)  and  other 
(iovernment  agencies  doing  development  work,  can  easily  arise  over  ambiguities 
or  contradictions  in  specifications,  SOWs,  and  other  contract  components. 

Thus,  Validation  Phase  design  and  analysis  should  continue  until  the  Program 
Office  clearly  understands  the  system  definition,  judges  it  desirable  and 


AFR  70-15,  paragraph  2-1a. 
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feasible,  and  agrees  that  it  has  been  precisely  documented.  Deciding  when 
system  definition  is  satisfactory  is  an  Important  Program  Office 
responsibility.  Higher  levels  of  authority  (e.g.,  the  DSARC)  cannot  be 
expected  to  detect  all  important  design  deficiencies. 

^•3-1  System  Design 

^.3. 1.1  Allocated  Baseline  Development.  The  Allocated  Baseline  (Table 
2,  Set  E)  is  prescribed  as  the  major  Validation  Phase  product.  Starting  with 
the  updated  Functional  Baseline  (Table  2,  Set  B),  Allocated  Baseline 
development  entails  verifying  or  changing  the  system's  first-level  design,  and 
extending  it  to  a third  level.  First-level  system  design  consists  of: 

a.  subdividing  the  system  defined  in  the  System  Specification  into  a 
number  of  components  called  Functional  Areas*; 

b.  specifying  the  interfaces  among  the  Functional  Areas; 

c.  allocating  among  these  Functional  Areas  the  system's  functional, 
technical  performance,  external  interface,  design,  and  testing 
requirements;  and 

d.  if  the  system  is  to  be  segmented  (see  Section  4.3.3),  identifying 
the  System  Segments,  and  the  Functional  Areas  that  belong  to  each 
Segment . 

Second-level  system  design  consists  of  similarly  dividing  each  Functional  Area 
into  a number  of  components  called  Configuration  Items  (CIs)**,  specifying 
their  interfaces,  and  allocating  the  system's  requirements  among  them. 

Software  CIs,  including  both  computer  programs  and  computer  data,  are  usually 
called  Computer  Program  Configuration  Items  (CPCIs).  However,  unless 
otherwise  qualified,  the  term  Configuration  Item  applies  to  both  equipment  and 
software.  Third-level  system  design  similarly  subdivides  each  Cl  into  parts 
called  Functions,  which  are  defined  in  its  Development  Specifications.# 

The  Allocated  Baseline  is  documented  in  a set  of  preliminary  Development 
Specifications,  one  per  Cl.  Also,  a correspondingly  revised  version  of  the 
System  Specification  must  be  developed.  AFR  800-14,  Vol.  II  (paragraphs  2-4  & 


* MIL-STD-480,  Configuration  Control  - Engineering  Changes.  Deviations,  and 
Waivers . paragraph  110.27. 

••  MIL-STD-480,  paragraph  110. 80;  MIL-STD-490,  paragraph  10. 3. 1;  and  MIL- 
STD-483(USAF) , paragraph  30.2. 

# MIL-STD-463(L'SAF) , paragraph  60.4.3.  The  Functions  of  a Cl  should  not  be 
confused  with  a system's  Functional  Areas  or  with  functional 
requirements;  i.e.,  the  definition  of  what  a system  (or  one  of  its  parts) 
must  do.  A software  developer  may  be  allowed  freedom  to  redefine  a 
CPCI's  parts  during  its  development.  When  such  redesign  occurs,  the 
finished  CPCI's  major  parts,  termed  Computer  Program  Components  (CPCs), 
may  differ  from  its  Functions  (see  Section  A4). 


Major  Validation  Phase  Activities 
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KEY  on  final  page. 

"None  apeclfleO"  means  that  no  directive  reviewed  specifies  a 
particular  form  for  this  product.  Hence,  memoranda  or  technical 
reports  would  be  used,  as  appropriate. 

Products'  DID  Numbers  are  bracketed  where  first  named. 
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Table  2 (Continued) 
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Table  2 (Concluded) 
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M Tb«  CovaiUMtit  roles  and  types  of  contractor  support  Indicated 
apply  to  the  entire  set  (e.g.,  Set  *,  Set  C),  or  to  parts  of  the 
set  (e.g.,  1.1),  with  the  description  of  which  they  line  up. 


2-5),  calls  this  the  Authenticated  System  Specification.  AFR  800-3, 
Engineering  for  Defense  Systems  (paragraph  4,b)  includes  the  Authenticated 
System  Specification  in  the  Allocated  baseline,  while  other  sources  exclude 
it.  (See,  e.g.,  MIL-STD-480,  paragraph  110.3  and  AFSCP  800-3,  paragraph  9-9). 
Given  this  choice  of  authorities,  this  guidebook  assumes  exclusion  of  the 
Authenticated  System  Specification  from  the  Allocated  Baseline,  consistent 
with  common  usage.  However,  system  development  should  be  based  on  both  the 
Authenticated  System  Specification  and  the  Development  Specifications,  and  on 
any  Segment  Specifications  (see  Section  M.3.3),  in  case  of  omissions  from,  or 
conflict  among,  the  Development  Specifications.  In  such  cases  the 
Authenticated  System  Specification  should  have  highest  precedence  (see  SOWG, 
Section  C2.5.1),  on  the  grounds  that  system-level  requirements  are  more 
fundamental  than  allocated  (i.e.,  derived)  requirements. 

^.3- 1-2  Configuration  Item  Definition.  The  number  and  composition  of  a 
system's  CIs  is  a critical  design  issue,  because  the  Government's  technical 
monitoring  activities  focus  mainly  on  CIs.  For  example,  each  CPCI  developed 
normally  requires  the  developer  to  prepare  an  individual  Computer  Program 
Product  Specification  (see  Section  A^l),  an  individual  Test  Plan,  and  related 
Test  Procedures.  Each  Cl  usually  undergoes  individual  design  reviews.  One  or 
more  WBS  Elements  (see  SOWG,  Appendix  A)  must  also  be  defined  for  each  Cl,  for 
use  in  cost  reporting  and  analysis. 

A system  of  many  CIs  has  many  formally  defined  interfaces.  The  separate 
reports,  other  documents,  and  other  monitoring  activities  required  can  support 
good  Government  visibility  into,  and  control  of,  the  development  process. 

However,  if  a system  is  partitioned  into  too  many  CIs,  the  large  number 
of  document  review,  Engineering  Change  Proposal  (ECP)  processing,  and  other 
monitoring  activities  entailed  may  fragment  insight  and  cause  excessive 
delays,  significantly  impeding  development  progress.  Independent  or 
sequential  Government  monitoring  of  individual  CIs  may  partly  Ignore  the  needs 
of  closely  related  CIs,  so  that  decisions  made  about  one  Cl  may  adversely 
affect  another.  Conducting  joint  design  reviews  for  the  members  of  each 
closely  related  set  of  CIs,  and  employing  the  same  Government  personnel  to 
monitor  all  the  set's  members,  can  improve  overall  visibility.  Nevertheless, 
even  thorough  design  review  rarely  prevents  subsequent  discovery  of  some 
necessary  changes  in  Cl  scope  or  external  Cl  interfaces.  Such  changes  require 
formal  ECP  preparation  and  Configuration  Control  Board  (CCB)  action  during 
development,  activities  that  typically  consume  weeks  or  months.  Largely 
because  of  its  greater  quantity  of  baselined  information  (e.g.,  inter-CI 
interface  definitions  in  Development  Specifications),  a multi-CI  system  may 
require  more  ECPs  during  its  development  than  a system  of  fewer  CIs. 

Similarly,  the  effort  needed  to  review  and  coordinate  revisions  to  Product 
Specifications,  Test  Plans,  Test  Procedures  and  other  required  documents 
depends  significantly  on  the  number  of  documents  reviewed  as  well  as  on  the 
scope  of  each.  Like  ECP  processing,  document  review  can  entail  long  elapsed 
times,  because  comments  must  typically  be  solicited  from  many  reviewers, 
formally  coordinated,  and  reflected  in  one  or  more  revisions  before  approval. 
Thus,  a multi-CI  system's  development  may  suffer  more  delay  from  Government 
monitoring  activities  than  a system  of  fewer  CIs. 
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Somewhat  different  problems  can  arise  if  a system's  CIs  are  few,  but  ill- 
defined.  This  situation  exists  to  the  extent  that  one  Cl  contains  processes 
that  interact  more  strongly  with  other  CIs  than  with  one  another.  A system  of 
ill-defined  CIs  is  most  likely  when  Cl  definition  occurs  hastily  without 
adequate  preliminary  design  and  design  validation  (see  Section  i».3.2).  Here 
the  inter-CI  interfaces,  although  few,  are  complex.  As  a result,  the  larger 
scope  of  the  individual  Cl  design  reviews  will  still  fail  to  spot  many 
inconsistencies  among  CIs.  Also,  the  complex  internal  workings  of  large,  ill- 
defined  CIs  discourages  learning  and  discovery  of  internal  flaws.  Both 
factors  encourage  overlooked  design  errors  during  document  study  and  design 
reviews.  These  oversights  lead  later  to  many  ECPs  and  to  progressively  more 
expensive  repairs,  depending  on  when  each  error  is  detected. 

We  know  of  no  well-defined  procedure  to  specify  an  optimum  set  of  CIs.  ■ 
However,  the  guidelines  stated  below  should  help  define  a good  set  of  CPCIs, 
although  they  are  incomplete. 

a.  Assign  processes  that  interact  strongly  (e.g.,  in  many  or  complex 
ways)  to  the  same  CPCI. 

b.  Assign  processes  that  have  little  or  no  interaction  to  different 
CPCIs. 

c.  Allocate  to  different  CPCIs  processes  that  will  execute  in  different 
computers . 

d.  Assign  to  different  CPCIs  processes  whose  development  can  feasibly 
be  finished  at  significantly  different  times,  if  such  phased 
development  will  expedite  overall  system  development. 

e.  Allocate  to  different  CPCIs  software  to  be  procured  separately. 

f.  Include  in  each  CPCI  no  more  than  a small,  well-knit  group  of 
Government  monitors  can  efficiently  track,  assuming  reasonable 
working  relationships  between  them  and  the  types  of  personnel  who 
will  manage  and  develop  the  CPCI. 

It  should  be  clear  that  applying  these  guidelines  entails  considerable 
preliminary  design  and  analysis.  Guidelines  a,  b,  d,  and  f also  apply  to 
equipment  CIs,  as  does  guideline  e if  ’’equipment"  is  substituted  for 
"software" . 

Even  when  a system  has  many  small  CIs,  WBS  definition  must  generally 
extend  below  the  Cl  level,  to  the  CPC  or  major  routine  level,  in  order  to 
yield  data  adequate  for  both  thorough  contractor  perfor.^ance  monitoring  and  to 
sound  future  software  cost  estimation.  Such  detailing  of  WBS  Elements  below 
the  Cl  level  is  best  done  by  the  development  organization,  with  Program  Office 
concurrence.  (See  SOWG,  Appendix  A for  explanation  of  WBSs).  Such  WBS 
Element  breakdown  should  be  done  as  the  detailed  design  of  each  Cl  unfolds, 
and  incorporated  in  the  Extended  Contract  WBS  (see  SOWG,  Section  A4.6). 
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'♦•3-1-3  Common  System  Definition  Errors.  One  common  error  in  system 
definition  is  i'ailure  to  specify  as  CPCIs  certain  essential  Support  Software 
(e.g.,  Executive,  equipment  and  software  diagnostics,  software  development  and 
maintenance  aids,  test  drivers,  test  data  generators,  data  collection  and  data 
reduction  programs)*.  As  a result,  the  Government  may  lack  normal  control  of 
and  visibility  into  this  software's  functional  & design  characteristics,  and 
may  even  lack  the  right  to  use  the  software  throughout  the  system's  lifetime. 
Such  rights  of  control,  visibility  and  permanent  use  can  be  critical;  e.g.,  to 
validating  test  results,  to  testing  Deployment  Phase  software  modifications. 

If  use  of  proprietary  Operational  or  Support  Software  is  planned,  the  CPDP 
(see  Section  H.4.5)  should  detail  its  use  in  the  system.  Furthermore,  the 
appropriate  contract  should  specifically  provide  for  delivery  of  that 
proprietary  software  with  satisfactory  documentation  and  rights  of  duplication 
& use  (see  SOWG,  Section  C2.5.4). 

Another  common  error  is  failure  to  prescribe  precisely  the  system's 
interfaces  with  its  operators  (e.g.,  terminal  users).  These  interfaces  should 
be  considered  requirements,  not  design  options,  because  a good  man-machine 
interface  is  quite  heavily  influenced  by  detailed  operational  requirements. 

Special  problems  may  arise  when  use  is  planned  of  existing  software 
(e.g.,  the  Executive,  a compiler,  diagnostics)  that  was  developed,  perhaps  for 
commercial  use,  independent  of  standard  Air  Force  configuration  control, 
testing  and  documentation  practices.  Although  incorporating  such  software, 
where  appropriate,  may  save  significant  development  time  and  cost,  this 
software  or  its  documentation  may  be  somewhat  deficient  for  the  intended  Air 
Force  application.  Thus,  during  the  Validation  Phase,  all  such  existing 
software  should  be  tested,  and  its  documentation  reviewed,  against  system 
requirements.  Plans  should  then  be  made  to  upgrade  or  augment  this  software 
and  its  documentation  during  the  Full-Scale  Development  Phase,  to  correct 
deficiencies.  For  example,  if  use  of  a commercially  available  Executive  is 
planned,  this  Executive  should  be  allocated  functional,  design,  interface, 
performance  and  test  requirements.  The  Executive  should  then  be  tested  for 
ability  to  satisfy  all  its  allocated  requirements.  Again,  the  Executive's 
documentation  should  be  reviewed  against  the  needs  of  the  planned  Air  Force 
system's  operators,  development  programmers,  and  maintenance  programmers  to 
assure  its  satisfactory  organization  and  content.  Existing  commercial 
documentation  need  not  ccnform  precisely  to  Air  Force  documentation  standards 
(e.g.,  for  Type  B5  and  Type  C5  specifications  per  MIL-STD-490  and  MIL-STD- 
4b3(USAF)).  However,  these  standards  should  be  reviewed  for  factors 
appropriate  to  Judging  existing  documentation  against  expected  needs.  Note 
that  the  Government  may  need  to  acquire  Limited  Rights  to  this  existing 
software,  and  Restricted  Rights  to  its  documentation  (see  SOWG,  Section 
C2.5.4)  in  order  to  use  or  upgrade  them. 


SOWG,  Table  A-3  identifies  many  such  types  of  Support  Software.  The 
Software  Acauialtion  Management  Guidebook;  Software  Developioent  and 
Maintenance  Facilities  discusses  typical  support  software  and  its  uses. 
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4.3.2  System  Design  Validation 


The  Validation  Phase  is  intended  to  develop  a low-risk  system 
design  clearly  able  to  meet  the  requirements  of  the  System  Specification, 
within  the  cost  and  schedule  thresholds  established  by  the  approved  version  of 
DCP  I.  Attaining  this  goal  will  usually  require  the  definition,  partial 
development,  and  evaluation  of  several  alternate  designs.  A typical  Major 
Defense  System's  complexity  makes  very  difficult  the  accurate  evaluation  of  a 
design  alternative,  especially  its  workload-handling  capacity  and  achievable 
response  times,  which  often  defy  precise  mathematical  analysis.  If,  as  is 
usual,  essentially  complete  prototypes  of  the  system  alternatives  are 
unavailable  for  instrumentation,  discrete  event  simulation  of  the  system 
alternatives  offers  the  best  chance  of  developing  sound  performance  prediction 
data,  if  based  on  well  understood  requirements,  sound  analytic  technique,  and 
realistic  estimates  of  workload,  component  size,  and  component  performance. 
Tnus,  the  simulation  computer  programs  developed  during  the  Conceptual  Phase 
(see  Table  1,  Set  J)  should  be  refined  and  used  during  the  Validation  Phase  to 
help  evaluate  the  design  alternatives.  If  not  yet  available,  these  programs 
snould  be  developed  during  the  Validation  Phase. 

A system  design  can  seldom  be  validated  unless  first  developed  in 
considerable  detail.  For  example,  to  show  that  a proposed  system  will  accept 
and  process  a particular  type  of  input,  and  produce  expected  output,  within 
prescribed  response  time  limits,  typically  involves  estimating  and  summing  the 
processing  times  of,  and  the  expected  queuing  delays  at,  each  system  component 
that  handles  these  outputs  and  their  precursors.  Considerable  detailed 
computer  program  design,  sizing,  estimation  of  routines'  execution  times,  and 
subsequent  simulation  may  be  essential  to  obtain  credible  estimates  of  the 
corresponding  processing  and  queue-residence  times.  The  CPCs  of  CPCIs  that 
implement  time-consuming  algorithms  are  prime  candidates  for  such  design, 
sizing,  estimation,  and  simulation. 

Whenever  properly  conducted  system  design  validation  results  in 
selection  of  a preferred  design  shown  able  to  meet  the  system's  requirements, 
this  design  should  not  be  discarded.  Developing  and  validating  such  a design 
requires  extensive  effort  during  the  Validation  Phase,  which  Offerers  for 
Full-Scale  Development  Phase  contracts  are  unlikely  to  duplicate.  Especially 
if  the  Government's  preferred  design  resulted  from  an  open  design  competition, 
there  is  little  chance  that  a Full-Scale  Development  Phase  Offerer  will 
suggest  a better  design,  and  considerable  risk  that  this  design  would  fail 
thorough  validation.  Instead,  the  Government's  validated  design  should  be 
incorporated  in  the  appropriate  Cl  Development  Specifications.  These 
specifications  should  Include  all  design  requirements  and  other  assumptions 
employed  in  validating  the  design.  However,  unvalidated  design  detail  should 
be  omitted  to  avoid  unnecessarily  constraining  Offerers'  design  freedom,  and 
also  because  it  may  be  wrong! 

Subsequently,  the  Government  may  let  Offerers  for  Full-Scale 
Development  Phase  contracts  propose  design  modifications.  However,  this 
approach  should  be  followed  only  if  the  Government  expects  a substantially 
improved  design  to  result,  and  if  time  and  other  resources  permit  proper 
evaluation  of  the  proposed  design  modifications.  If  proposing  design  changes 
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is  allowed,  each  Offerer  should  be  required  to  submit  all  evidence  necessary 
for  Government  validation  that  his  proposed  design  will  satisfy  the  system 
requirements  better  than  the  Government-specified  design.  The  Government  has 
no  obligation  to  accept  a proposed  design.  Indeed,  unless  an  Offerer  can 
prove  that  his  proposed  system  design  changes  can  better  meet  the  system's 
requirements,  the  Government  should  not  accept  the  proposed  changes,  during  or 
after  contract  negotiations.  Any  acceptable  proof  should  meet  the  same 
standards  used  to  select  the  Allocated  Baseline  developed  during  the 
Validation  Phase. 

Some  consider  imposing  a validated  design  on  a contractor  unsound 
because  it  would  limit  his  design  freedom  and  might  thus  preclude  a better 
system  design.  However,  proper  design  validation  will  yield  a sound,  low-risk 
design,  while  the  risk  of  unsuccessful  development  based  on  an  unvalidated 
design  is  much  greater.  Considering  the  usually  severe  adverse  affects  of 
unsuccessful  system  development,  it  will  rarely  pay  to  select  a high-risk 
design  over  a validated  low-risk  design. 

Others  maintain  that  imposing  a design,  validated  or  not, 
eliminates  contractor  responsibility  for  developing  a defective  system.  This 
need  not  be  true.  A contractor  who  signs  a properly  worded  contract  accepts 
legal  liability  for  developing  a system  that  meets  its  specified  performance 
requirements  subject  to  its  specified  design  constraints.  More  important, 
regardless  of  legal  liability,  the  Government  retains  the  main  risks 
associated  with  development  of  a Major  Defense  System.  These  include  the 
practical  difficulties  of  recovering  sunk  development  costs,  the  high  costs  of 
system  modification  (or  redevelopment),  and  the  operational  Impact  of  late 
delivery  and  reduced  system  capability.  The  Government  should  thus  insist  on 
a properly  validated  design  to  reduce  such  risks. 

Demonstration  of  feasibility  should  include  building  and 
evaluating  experimental  equipment  and  software  for  any  parts  of  the  system 
deemed  especially  critical  or  risky  during  Conceptual  Phase  analyses. 
Evaluation  of  this  software  and  equipment  should  assess  both  design  and 
performance,  and  results  should  be  factored  into  other  Validation  Phase  effort 
(e.g.,  simulation).  Equipment  evaluation  should  also  encompass  reliability, 
maintainability  and  produclbility . Note  that  evaluating  this  equipment  and 
software  may  both  entail  developing  automated  evaluation  aids.  Such  aids 
include  software  to  generate  and  present  test  data,  to  trace  execution 
sequences,  to  help  measure  elapsed  times,  to  record  results,  and  to  control 
test  sequencing.  Developing  experimental  equipment  and  software  and  their 
evaluation  aids  should  start  during  the  Conceptual  Phase  (see  Table  1,  Set  G) , 
because  of  long  lead  times.  The  magnitude  of  the  effort  necessary  may  require 
contractor  support. 

'i.S-S  System  Segment  Definition 

As  the  Allocated  Baseline  evolves,  it  may  seem  desirable  to  divide 
the  system  into  two  or  more  major  parts  for  development  by  separate 
organizations  during  the  Full-Scale  Development  Phase.  Each  such  part,  which 
M1L-STD-483(USAF)  (paragraph  30.6.2)  terms  a System  Segment  (or  Segment  for 
short),  consists  of  one  or  more  complete  Functional  Areas  and  usually  includes 
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several  CIs.  One  possible  good  reason  to  segment  a system  is  that  essential 
expertise  applicable  to  different  parts  of  the  system  may  be  split  among 
different  potential  contractors.  For  example,  a separate  Software  Segment 
might  be  defined  to  encourage  participation  in  the  system  development  by 
software  development  firms  better  qualified  than  general  system  contractors  to 
produce  critical  software. 

However,  segmentation  introduces  an  additional  configuration 
management  level  between  the  system  level  and  the  Cl  level.  Thus, 
segmentation  increases  the  complexity  of  system  management,  introduces 
additional  costs,  and  may  cause  more  problems  than  it  resolves.  For  example, 
each  Segment  must  be  allocated  a subset  of  the  system's  requirements, 
including  test  requirements.  Once  contractually  defined,  such  allocations  are 
hard  to  change,  and  if  not  well  conceived  can  cause  severe  performance 
problems  or  disputes  about  responsibilities.  MIL-STD-^t83(USAF)  (paragraph 
30.6),  requires  Segment  Specifications  (see  Section  A2)  in  some  cases.  If  so 
they  must  be  prepared  and  reviewed.  Per  MIL-STD-483( USAF) , paragraph  20, 
intersegment  interfaces  must  be  defined  in  Interface  Control  Drawings  (ICDs), 
and  an  Interface  Control  Working  Group  (ICWG)  must  be  established  to 
adjudicate  disputes  and  changes  related  to  Interfaces  between  Segments.  ICD 
revisions,  typically  Engineering  Change  Orders  (ECOs),  must  be  approved  by  the 
ICWG.  If  ICWG  actions  are  not  closely  coordinated  with  CCB  actions,  they  may 
cause  inconsistencies  in  the  system  baseline.  Similarly,  the  CCB  evalt  tion 
of  an  ECP  may  involve  several  contractors  and  require  coordination  witi,  ‘he 
ICWG.  Segment-level  requirements  reviews,  design  reviews  and  tests  musk  le 
planned,  and  these  activities  must  be  monitored.  The  Segment  Tests,  the  JG, 

the  ICDs  and  Segment  Specifications  are  not  required  if  a system  is  not 
segmented . 

4.4  Validation  Phase  Planning  Activities 

Refer  to  Table  2,  Sets  C,  D,  F & G.  As  discussed  subsequently,  several 
versions  of  some  of  the  plans  mentioned  (e.g.,  the  PMP)  may  be  prepared  in 
different  Acquisition  Life  Cycle  phases,  or  a plan's  development  may  sometimes 
begin  in  the  Conceptual  Phase.  Despite  these  variations,  preparation  of  these 
plans  are  most  appropriately  discussed  as  Validation  Phase  activities. 

To  encourage  alternative  designs  and  sound  analysis,  two  or  more  parallel 
Validation  Phase  contracts  may  be  let  for  system  (or  Segment)  design, 
analysis,  and  planning  for  subsequent  Acquisition  Life  Cycle  phases.  The 
stimulus  of  competition  is  a major  aim  of  this  approach.  Hence,  provision 
should  be  made  to  award  the  winning  Validation  Phase  contractor(s)  major 
development  roles  during  the  Full-Scale  Development  Phase,  provided  a 
favorable  Ratification  Decision  is  made.  Contractor  selection  for  Full-Scale 
Development  Phase  work  will  depend  primarily  on  the  competitors'  cost  and 
schedule  estimates  and  on  Government  assessment  of  their  management  skills,  as 
well  as  on  the  proposals'  technical  merit.  Under  these  circumstances,  each 
Validation  Phase  RFP  (i.e.,  one  per  planned  contract)  should  prescribe 
preparation  and  delivery  of  a draft  CPDP  and  (if  the  contract  involves  System 
Engineering  effort)  a SEMP,  for  the  Full-Scale  Development  Phase.  During 
Validation  Phase  Source  Selection  (Set  D),  Government  personnel  must  review 
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the  version  of  each  such  plan  produced  by  each  prospective  contractor  or 
Government  development  organization. 

Besides  Validation  Phase  RFP  review,  and  Validation  Phase  Source 
Selection,  the  other  chief  management-oriented  Validation  Phase  activities  to 
be  performed  by  Government  personnel  include: 

a.  revising  the  Program  Management  Plan  (PMP),  first  prepared  during 
the  Conceptual  Phase,  to  reflect  guidance  in  PMD  and  AFSC  Form  56 
supplements ; 

b.  developing  a first  version  of  the  Computer  Resources  Integrated 
Support  Plan  (CRISP); 

c.  writing  a Test  and  Evaluation  Master  Plan  (TEMP); 

d.  drafting  a Training  Plan;  and 

e.  preparing  the  initial  draft  DCP  II  and  related  backup  material. 

In  addition.  Government  personnel  must  produce  the  draft  RFP(s)  for  the  Full- 
Scale  Development  Phase  contract(s).  The  RFP  for  each  contract  involving 
System  Engineering  effort  should  require  a SEMP,  and  the  RFP  for  each  contract 
Involving  software  development  should  require  a CPDP,  for  the  reasons  stated 
in  the  previous  paragraph  about  competitive  Validation  Phase  RFPs. 

^.U.1  The  Program  Management  Plan  (PMP) 

AFR  800-2  prescribes  the  PMP.  AFSCP  800-3  (especially  Attachments 
3 & 4)  further  defines  it.  Basically,  the  PMP  must  describe  the  system  to  be 
acquired,  identify  available  resources,  define  the  overall  acquisition 
management  approach,  identify  the  participating 'Government  organizations,  and 
specify  their  roles.  Topics  to  be  addressed  include:  Program  Summary  & 

Authorization,  Intelligence,  Program  Management,  System  Engineering,  Test  & 
Evaluation,  Communications/Electronics,  Operations,  Civil  Engineering, 
Logistics,  Manpower  & Organization,  Personnel  Training,  Security,  and 
Directives  Application.  The  PMP,  prepared  by  the  Program  Office,  requires 
coordination  by  all  participating  commands.  An  initial  version  of  the  PMP  is 
normally  prepared  in  response  to  the  Conceptual  Phase  PMD.  This  must  be 
revised  to  reflect  supplementary  direction.  Major  revision  in  response  to 
Validation  Phase  PMD  and  AFSC  Form  56  direction  should  be  expected. 

The  Computer  Resources  Integrated  Support  Plan  (CRISP) 

Prescribed  by  AFR  800-14,  Vol.  II,  for  acquisitions  that  Involve 
Computer  Resources  (i.e.,  computer  equipment,  software,  related  documentation, 
and  associated  personnel),  the  CRISP  is  Intended  to  clarify  the  software- 
related  roles  of  the  Government  participants  in  a system's  development.  The 
CRISP 
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"identifies  organizational  relationships  and  responsibilities  for  the 
management  and  technical  support  of  computer  resources.  It  functions 
during  the  full-scale  development  phase  to  identify  computer  resources 
necessary  to  support  computer  programs  after  transfer  of  program 
management  responsibility  and  system  turnover."* 

AFR  800- 1U,  Vol.  II  (paragraph  3-10)  directs  formation  of  a Computer  Resources 
Working  Group  (CRWG),  chaired  by  the  Program  Office  until  Program  Management 
Responsibility  Transfer  (PMRT)  and  System/Equipment  Turnover  (see  Section 
6.4).  The  CRWG  is  responsible  for  initial  development  and  subsequent  updating 
of  tne  CRISP.  Although  not  required,  development  during  the  Conceptual  Phase 
of  the  portions  of  the  initial  CRISP  that  reflect  the  Government  participants' 
overall  roles  and  missions  could  aid  system  planning. 

4.4.3  The  Test  and  Evaluation  Master  Plan  (TEMP) 

Per  AFR  800-14,  Vol.  II  (paragraphs  5-2  and  5-5),  the  TEMP  is 
intended  to  supplement  the  PMP  and  the  Test  & Evaluation  Objectives  Annex 
(TEOA)  of  the  PMD.  Per  AFR  80-14,  Test  and  Evaluation.  AFSC  Supplement  1 
(paragraph  20e),  the  TEMP  is  Intended  to 

"document  a coordinated  position  for  all  the  participants  in  the  T4E  of  a 
particular  program,  and  give  decision  makers  an  opportunity  to  examine 
the  plan  for  accomplishing  T&E". 

TEMP  development  is  a Program  Office  responsibility,  but  the 
Operating  Command  and  Supporting  Command,  plus  any  other  agencies  involved  in 
the  system's  Test  and  Evaluation  (T4E)  must  coordinate  on  it.  The  TEMP  must 
address: 


a.  critical  questions  and  areas  of  risk; 

t.  test  objectives; 

c.  the  T&E  program  outline; 

d.  responsibilities  of  all  participants.  Including 
contractors ; 

e.  test  costs  and  schedules;  and 

g.  needed  test  resources  (e.g..  Instrumentation,  other  equipment, 
facilities,  data). 

AFSC  Supplement  1 excludes  Follow-on  Operational  T&E  (FOT&E)  from 
the  temp's  scope.  AFR  80-14  does  not.  They  agree  that  the  TEMP  must 
encompass  Development  Test  & Evaluation  (DT&E)  and  Initial  Operational  Test  & 
Evaluation  (lOT&E).  DT&E,  an  Implementing  Command  responsibility,  includes 
all  formal  CPCI  testing  (e.g.,  all  Formal  Qualification  Tests  (FQT)),  Segment 
testing  (if  any),  and  system-level  testing  against  System  Specification 


AFR  800-14,  Vol.  II,  paragraph  3-8. 
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requirements.  Operational  Test  and  Evaluation  (0T4E)  (i.e.,  lOT&E  + F0T4E)  is 
intended  mainly  to  assess  a Major  Defense  System's  operational  utility,  in 
contrast  to  its  formal  compliance  with  specifications.  0T4E  is  normally  the 
responsibility  of  the  Using  Command,  or  of  the  Air  Force  Test  and  Evaluation 
Center  (AFTEC),  with  assistance  from  the  Implementing  Command  and  the 
Supporting  Command  (usually  the  Air  Force  Logistics  Command  (AFLC)). 

Per  AFR  80-1^  (paragraph  20e)  the  TEMP  must  be  prepared  "as  early 
as  possible. . .prior  to  initiation  of  full-scale  development."  Normally  the 
TEMP  will  be  prepared  early  in  the  Validation  Phase,  but  under  some 
circumstances  could  be  drafted  earlier.  The  TEMP  must  be  updated  to  reflect 
each  significant  change  in  the  test  program. 

The  Training  Plan 

Per  AFSCP  600-3  (paragraph  3),  the  Training  Plan  is  intended  to 
establish  requirements  for  training  Air  Force  personnel  in  the  operation  and 
maintenance  of  the  system,  beginning  during  Full-Scale  Development.  Training 
Plan  preparation  entails  active  participation  by  Implementing  Command,  Using 
Command,  Supporting  Command,  and  Air  Training  Command  (ATC)  personnel. 

. 5 The  Computer  Program  Development  Plan  (CPDP) 

AFR  800-14,  Vol . II  (paragraphs  3-5  and  3-9),  requires  a CPDP  for 
every  Major  Defense  System  acquisition  that  includes  software  development,  and 
prescribes  CPDP  contents.  In  part,  the  CPDP  must  state:  the  organization  and 
responsibilities  of  the  software  development  group(s);  the  skill  level  of  the 
software  design,  development  4 maintenance  personnel;  software  management  and 
technical  control  methods;  software  Quality  Assurance  (QA)  methodology; 
software  development  schedule  and  milestones;  configuration  control  and  status 
monitoring  procedures;  documentation  and  training  methods;  and  programming 
standards.  CPDP  preparation  is  an  Implementing  Command  responsibility. 
However,  each  of  tne  Offerers  for  a software  development  contract  (and  each  of 
any  prospective  Government  software  development  organizations)  should  be 
required  to  prepare  a CPDP  as  part  of  its  proposal;  a CPDP  will  provide 
important  evidence  of  its  Offerer's  competence.  Such  a CPDP  should  be 
prepared  regardless  of  Acquisition  Life  Cycle  phase.  For  example,  a RFP 
issued  during  the  Conceptual  Phase  that  calls  for  extensive  software 
development  should  require  each  Offerer  to  submit  a CPDP  as  part  of  his 
proposal.  After  modification  during  negotiation  of  contracts  (or  interagency 
memoranda  of  agreement)  the  CPDP  of  each  selected  contractor  or  Government 
development  organization  should  become  part  of  its  agreement  with  the 
Implementing  Command,  so  that  its  provisions  can  be  enforced.  The  CPDP  will 
probably  require  updating  during  development;  e.g.,  to  reflect  schedule 
changes.  Thus,  a SOW  task  should  provide  for  such  updating. 

4.4.6  The  System  Engineering  Management  Plan  (SEMP) 

MIL-STD-499A,  Engineering  Management,  and  AFR  800-3,  define 
Systems  Engineering,  and  prescribe  the  Systems  Engineering  effort  needed 
during  the  acquisition  of  Major  Defense  Systems.  They  require  the  development 


of  a three-part  SEMP  and  prescribe  Its  contents*.  If  a planned  contract  Is  to 
include  Systems  Engineering  effort,  the  RFP  for  that  contract  should  require  a 
SEMP  as  part  of  each  Offerer's  proposal,  to  become  binding  (after  possible 
negotiated  change)  upon  contract  award.**  A SOW  task  should  provide  for  the 
SEMP's  subsequent  updating. 

Jj.  5 Termination 

Refer  Table  2,  Set  H.  A second  DSARC  review  and  the  Ratification 
Decision  are  prescribed  to  terminate  the  Validation  Phase,  in  order  to  judge 
the  adequacy  of  its  results  and  to  reassess  the  continued  importance  of 
developing  the  planned  system.  An  adverse  Ratification  Decision,  which  would 
cause  program  termination,  could  result  from  any  of  the  following: 

a.  Inadequate  Validation  Phase  products; 

b.  detection  of  severe  and  evidently  Insurmountable  technical  problems 
during  the  Validation  Phase; 

c.  excessively  escalating  costs;  or 

d.  sufficient  reduction  in  the  operational  need  for  the  planned  system. 


MIL-STD-499A,  paragraph  5.1. 
AFR  800-3,  paragraph  i|.c. 
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5. 


FULL-SCALE  LEVELOPMENT  PHASE 


5. 1 Objectives 

The  Full-Scale  Development  Phase  is  intended  to  yield: 

a.  a working  prototype  of  the  (lajor  Defense  System  (or  the  system,  if 

there  are  to  be  no  replicas);  *- 

b.  test  j’esults  proving  that  this  prototype  can  meet  its  functional  and 
performance  requirements; 

c.  a cadre  trained  in  the  system's  operation  and  maintenance;  and 

d.  the  documentation  needed  to  begin  the  system's  Production  Phase  (if 
any),  or  otherwise  needed  for  its  Deployment  Phase. 

These  objectives  entail  completing  the  system's  engineering  design;  resolving 
all  major  uncertainties,  outstanding  issues,  and  other  problems;  and 
thoroughly  testing  the  functions  and  performance  of  the  prototype  system  and 
its  components.  Note  that  for  the  system's  software,  the  Full-Scale 
Development  Phase  is  intended  to  yield  the  initial  operational  versions  of  the 
Computer  Programs,  not  prototypes.  "Prototype"  is  properly  applied  to 
preproduction  equipment  whose  form,  fit  cind  function  will  be  identical  to 
those  of  the  (multiple)  production  units  planned,  but  which  may  differ  from 
the  production  units  in  other  ways. 

5.2  Initiating  Events 

See  Table  3.  Set  A.  A favorable  Ratification  Decision  by  the  Secretary 
of  Defense  begins  the  Full-Scale  Development  Phase.  The  Ratification  Decision 
may  prescribe  redirection  of  certain  system  goals,  schedules,  allowable  costs, 
and  other  constraints.  Both  Headquarters,  USAF,  and  AFSC  may  issue 
supplementary  guidance.  All  are  then  to  be  reflected  in  a revised  PMD.* 

5. 3 Other  Activities  and  Related  Products 

See  Table  3,  Sets  B through  G.  Full-Scale  Development  Phase  work  is 
treated  less  fully  than  the  Conceptual  Phase  and  Validation  Phase  activities, 
discussed  in  Sections  3 and  4,  because  model  Full-Scale  Development  Phase  SOW 
paragraphs  and  commentary  on  them  are  included  in  the  SOW  guidebook.  However, 
several  major  points  about  certain  Full-Scale  Deveiopment  Phase  activities  and 
products  should  be  noted. 


AFR  70-15,  paragraph  2-1b. 
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Major  Full-Scale  Developn»ent  Phase  Activities 
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S««  Key  on  final  page. 

■Nona  apeclfiad"  aeana  that  no  directive  reviewed  speclflee  a 
particular  fora  for  this  product.  Hence,  aeaoranda  or  technical 
reporta  ahould  be  used  as  appropriate.  , 

Products'  DID  nuabers  are  bracketed  when  first  naaed. 
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Table  3 (Continued) 
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First,  Table  3i  Set  B,  assumes  that  Full-Scale  Development  Phase  RFP 
issuance,  Source  Selection,  and  contract  award  will  occur  early  in  the  Full- 
Scale  Development  Phase,  per  planning,  development  work  allocation,  and  Full- 
Scale  Development  Phase  SOW  draft  preparation' during  the  Validation  Phase. 
However,  contracts  negotiated  for  Validation  Phase  work  might  instead  also 
provide  for  Full-Scale  Development  Phase  effort,  at  the  Government's  option. 
This  alternative  contractual  approach  could  eliminate  the  costs  and  delay  of 
separate  Full-Scale  Development  Phase  Source  Selection,  provided  no  drastic 
changes  to  the  defined  Validation  Phase  contract  options  were  necessary  when 
Full-Scale  Development  began.  Instead  of  a new  Source  Selection,  the 
acceptable  and  necessary  contract  changes  could  be  implemented  in  Supplemental 
Agreements  (SAs).  This  approach  would  also  facilitate  selection  (for  Full- 
Scale  Development  Phase  work)  of  any  Validation  Phase  competition  winner(s). 

Second,  significant  changes  and  further  detailing  of  both  management 
plans  and  system  design  normally  result  from  Validation  Phase  work.  (See 
Table  3.  Sets  C,  D,  and  E).  For  example,  preliminary  CPCI  Test  Plans  prepared 
during  the  Validation  Phase  may  require  revision  as  a result  of  Full-Scale 
Development  Phase  contract  negotiations.  Each  Full-Scale  Development  Phase 
contract  (new  or  SA)  should  incorporate  the  appropriate  changes  in  the  form  of 
a revised  Allocated  Baseline,  any  appropriate  Segment  Specification,  an 
Authenticated  System  Specification,  SOW  provisions,  and  related  CDRL  entries. 

Third,  the  system's  Operational  Software  (i.e.,  the  Executive(s)  and  the 
Application  Programs  necessary  to  meet  the  system's  operational  requirements), 
plus  the  Support  Software  necessary  to  build  and  maintain  the  Operational 
Software  and  to  support  DT&E  and  lOTiE,  must  normally  all  be  completed  during 
Full-Scale  Development.  For  both  Operational  Software  and  Support  Software 
developed  for  the  system,  such  completion  should  include: 

a.  successful  conclusion,  for  all  CPCIs , of  Preliminary  Design  Reviews 
(PDRs),  Critical  Design  Reviews  (CDRs),  FQTs,  Functional 
Configuration  Audits  (FCAs),  Physical  Configuration  Audits  (PCAs), 
and  Formal  Qualification  Reviews  (FQRs);* 

b.  successful  incorporation  of  all  CPCI  changes  necessary  to  complete 
satisfactorily  all  Segment-level  and  system-level  DT&E  requirements; 
and 

c.  delivery  of  all  approved  Computer  Program  Product  Specifications. 

In  contrast,  off-the-shelf  software  may  present  special  problems.  If 
proprietary  software  is  to  be  incorporated  in  the  system,  the  Government 
should  negotiate,  for  a reasonable  price.  Restricted  Rights  to  such  computer 
programs  and  Limited  Rights  to  their  documentation  (see  SCWG,  Section  C2.5.*0- 


MIL-STD-152KUSAF) , Technical  Reviews  and  Audits  for  Systems.  Equipment, 
and  Computer  Programs,  explains  PDR,  CDR,  FCA,  PCA,  FQR,  SDR,  etc.  Their 
application  to  software  acquisition  is  discussed  in  ESD-TR-75-85*  An  Air 
force  Guide  for  Monitoring  and  Reporting  Software  Development  Status. 


If  ttiese  negotiations  fail,  the  Government  should  buy  or  build  alternative 
software  with  satisfactory  documentation,  or  if  feasible  should  contract  for 
such  documentation.  Again,  off-the-shelf  software,  proprietary  or  otherwise, 
may  lack  adequate  documentation.  Software  Acquisition  Management  Guidebook: 
Software  Development  and  Maintenance  Facilities  explains  and  Illustrates 
typical  problems  that  can  result  when  Support  Software  is  acquired  with 
inadequate  rights  or  documentation.  If  so,  this  documentation  should  be 
supplemented  or  upgraded  to  support  adequately  its  use  and  maintenance  by 
Government  personnel.  If  this  is  Infeasible,  the  Government  should  acquire 
alternative  software  with  adequate  documentation.  Adequate  off-the-shelf 
software  need  not  meet  Standard  Computer  Program  Product  Specification  format 
requirements  (see  Section  A4),  but  should  contain  equivalent  information  in 
readily  usable  forms. 

Fourth,  preparation  and  updating  of  system-level  design  documentation  is 
assumed  to  begin  early  in  the  Full-Scale  Development  Phase  and  to  continue  at 
least  through  system-level  DT4E  (see  Table  3,  Sets  E.2  and  F.2).  This 
documentation  should  include  system-wide  equipment  and  software  block 
diagrams,  and  overview  descriptions,  keyed  to  relevant  Engineering  Drawings 
and  to  paragraphs,  figures  and  tables  in  the  system's  Authenticated  System 
Specification,  any  Segment  Specifications,  Cl  Development  Specifications,  and 
Cl  Product  Specifications.  Although  not  prescribed  by  stundard  Data  Item 
Descriptions,  such  documentation  can  significantly  help  in  training  new 
personnel,  in  detecting  incompatibilities  among  CIs , in  defining  and 
evaluating  EC^s,  in  defining  system-level  test  procedures,  and  in  interpreting 
system-level  test  results.  Also  presumed  are  at  least  two  additional  System 
Design  Reviews  (SDRs)  (see  Table  3i  Set  F.2),  conducted:  (1)  immediately 

following  all  Cl  CDRs;  and  (2)  after  all  Cl  FCAs,  after  all  CPCI  PCAs,  and 
before  system-level  DT&E.  The  first  of  these  additional  SDRs  should  show  that 
the  system  design,  as  represented  by  all  the  CIs'  designs,  is  complete, 
consistent,  and  able  to  meet  all  the  system  specification's  requirements.  The 
second  additional  SDR  should  show  that  the  system  is  ready  for  system-level 
testing.  This  SDR  should  show  that  the  system's  CIs,  as  modified  during  their 
development . are  complete  enough,  are  consistent  enough  with  one  another,  and 
well  enough  reflect  system-level  requirements,  to  assure  efficient  and 
successful  system-level  testing. 

Fifth,  Government-required  Preliminary  Qualification  Testing  (PQT)  (Table 
3,  Set  F.1)  should  be  minimized  or  eliminated.  The  resources  saved  might  be 
allocated  more  efficiently  to  better  PQT. 

5.4  Terminating  Events 

Refer  to  Table  3»  Set  H.  DSARC  review,  possible  modification,  and 
coordination  of  DCP  III  precede  the  Secretary  of  Defense's  Review  Decision, 
which  terminates  Full-Scale  Development.  Like  earlier  decisions,  the  Review 
Decision  may  terminate  the  program,  may  redirect  it,  or  may  allow  it  to 
proceed  as  planned  into  the  Production  and  Deployment  Phases.  Systems  whose 
equipment  requires  no  replication  may  skip  the  Production  Phase. 
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6. 


PRODUCTION  AND  DEPLOYMENT  PHASES 


6. 1 Ob  iectives 

The  prime  objective  of  the  Production  Phase  is  to  produce  and  install  in 
good  working  order  all  planned  replicas  of  the  Major  Defense  System.  The 
chief  goal  of  the  Deployment  Phase  is  to  use  the  system  effectively,  which 
entails  maintaining  it  efficiently  until  it  is  replaced  or  legitimately 
consumed  (e.g.,  during  warfare). 

6.2  Initiating  Events 

A favorable  Review  Decision  begins  the  Production  Phase,  or  the 
Deployment  Phase  Instead  if  the  Production  Phase  is  skipped.  This  can  happen 
if  there  are  to  be  no  replicas  of  the  Major  Defense  System,  and  if  the  Full- 
Scale  Development  Phase  result  is  operationally  acceptable.  The  Review 
Decision,  which  may  include  redirection  (e.g.,  of  quantities,  of  cost  and 
schedule  thresholds),  is  transmitted  via  Air  Staff  and  AFSC  channels. 
Supplementary  direction  (e.g.,  revised  AFSC  Budget  Authorization/Program 
Authorization  (BA/PA))  may  result.  PMRT  and  Turnover,  discussed  below, 
separate  the  Production  and  Deployment  Phases.  If  there  is  no  Production 
Phase,  PMRT  and  Turnover  normally  occur  shortly  after  the  Review  Decision. 

6.3  Other  Major  Activities  and  Events 

As  noted  in  Section  5.3,  all  of  the  Major  Defense  System's  Operational 
Software  and  most  or  all  of  its  Support  Software  must  normally  be  completed  by 
the  end  of  Full-Scale  Development.  "Production"  of  this  software  typically 
consists  of  copying  its  machine-readable  storage  media  (e.g.,  magnetic  tapes) 
and  of  reproducing  its  documentation,  both  trivial  types  of  operation.  One 
complication  is  a possible  need  to  adapt  each  copy  of  certain  software  by 
introducing  and  testing  site-specific  parameters.  Proper  software  design  will 
have  minimized  the  Impact  of  this  problem  by  concentrating  such  site-peculiar 
modifications:  e.g.,  in  a single  computer  data  base.  However,  to  the  extent 
that  Site  Adaptation  involves  developing  or  maintaining  different  software 
Versions,  these  Versions  should  be  produced  for  and  controlled  by  a single 
organization . 

Aside  from  Site  Adaptation,  software-related  Production  Phase  and 
Deployment  Phase  work  should  be  limited  mainly  to  maintenance  and  modification 
of  already  complete  software.  Although  development  of  some  new  software  for 
such  purposes  as  FOT&E,  exercises,  and  extensive  training  may  be  required, 
most  such  software  should  be  developed  during  the  Full-Scale  Development 
Phase,  since  it  must  be  Integrated  closely  with  the  Operational  Software. 

Software  maintenance  consists  of  investigating  alleged  software  errors 
and  devising  corrections  or  work-arounds  if  needed.  Software  modification 
involves  altering  software  to  support  changed  operational  system  requirements, 
or  making  desired  Improvements.  Both  may  Involve  testing  of  changes  at  each 
site  where  they  are  introduced. 
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Typically,  level-of-effort  contracts  are  let  for  both  of  these  functions, 
by  the  Using  Command  or  by  the  Supporting  Command.  Also  typically,  software 
maintenance  and  modification  are  managed  quite  Informally.  Instead,  It  Is 
recommended  that  software  maintenance  and  modification  contracts  Include  SOWs, 
CDRLs,  Delivery  Schedules  and  other  provisions  that  clearly  define  the 
appropriate  activities,  products.  Periods  of  Performance  and  financial 
controls,  as  these  are  defined  for  software  development.  Too  informal  an 
arrangement  can  obscure  assessment  of  progress  and  value  received. 
Alternatively,  Air  Force  instead  of  contractor  personnel  may  do  some  or  all  of 
the  software  maintenance  and  modification.  This  approach  is  termed  Organic 
Maintenance,  and  the  Support  Software  It  requires  should  be  covered  by  Full- 
Scale  Development  Phase  SOW  provisions. 

6.4  TerminatlnR  Events 

PMRT  from  the  Implementing  Command  to  the  Using  Command,  and  Turnover  of 
responsibility  for  support  of  the  system,  equipment  and  software  to  the 
Supporting  Command  (or  to  a Using  Command/Supporting  Command  combination) 
terminate  any  Production  Phase.  If  no  Production  Phase  is  planned,  these 
events  occur  shortly  after  a favorable  Review  Decision.  Their  occurrence 
marks  the  start  of  the  Deployment  Phase,  which  lasts  until  the  Major  Defense 
System  is  deactivated  or  replaced. 

After  PMRT  and  Turnover  the  chairmanship  and  membership  of  the  CRWG 
(which  is  responsible  for  preparation  and  updating  of  the  CRISP)  change,  per 
agreement  between  the  Using  Command  and  the  Supporting  Command.  Normally  the 
Supporting  Command  then  assumes  CRWG  chairmanship,  per  AFR  800-14,  Vol.  II 
(paragraph  3-10). 
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7. 


LESS  ELABORATE  ACQUISITIONS 


Acquisitions  that  do  not  satisfy  the  criteria  for  Major  Defense  Systems 
(stated  in  Section  2)  are  classified  as  Less-Than-Ma jor  Systems.  These  may  be 
managed  less  elaborately  than  Major  Defense  Systems.  However,  the  same 
"...management  principles. . .are  applicable  to  all  programs".*  Most  systems 
consisting  only  of  software,  and  many  smaller  systems  that  Include  both 
equipment  and  software,  would  fall  to  qualify  as  Major  Defense  Systems,  at 
least  on  grounds  of  predicted  costs.  In  paragraph  1-6,  "Management  of  Smaller 
Systems",  AFSCP  800-3  treats  the  major  differences.  These  pertain  mainly  to 
the  types  of  organization,  levels'of  revi-ew  & approval,  and  documentation 
required.  For  example,  Less-Than-Major  Systems  involve  neither  DCP 
preparation,  DSARC  review,  nor  Secretary  of  Defense  decision-making.  This 
information  appears  useful,  although  rather  general.  'AESCR  70-9,  Source 
Selection  Procedures,  and  AFSCR  80-15,  R&D  Source  Selection  Policy  and 
Guidance . provide  somewhat  more  specific  Information  about  Source  Selection 
for  certain  Less-Than-MaJor  Systems.  Unfortunately,  other  more  definitive 
directives  are  not  available,  and  only  general  guidance  can  be  given  here. 

In  planning  software  acquisition  for  Less-Than-Major  Systems,  a balance 
must  be  struck  between  the  benefits  and  the  high  costs  of  the  elaborate 
contract  monitoring  methods  typically  applied  in  Major  Defense  System 
acquisitions.  Unless  the  software  is  extremely  simple,  little  can  be 
eliminated  without  risking  misunderstood  requirements,  incorrect  operation, 
and  loss  of  Organic  Maintenance  capability.  Thus,  the  scope  of  specifications 
and  users'  manuals,  and  the  planning,  conduct  & reporting  of  tests,  cannot  be 
greatly  reduced  without  serious  risk  of  degraded  product  quality.  However, 
some  of  the  formal  procedures  involved  in  baselining,  reviewing  and  status 
monitoring  offer  opportunities  for  streamlining,  providing  these  controls' 
basic  objectives  are  not  thereby  compromised.  The  success  of  a more  informal 
approach  to  software  acquisition  management  may  also  depend  heavily  on  vesting 
responsibility  for  monitoring  contractor  progress  in  a few  competent  and 
dedicated  Government  personnel.  Such  success  may  also  depend  on  writing 
contracts  to  withhold  substantial  payment  until  the  Government  certifies 
satisfaction  with  the  product. 

One  reasonable  basis  for  preparation  of  procurement  documents  (e.g.,  a 
sow)  for  Less-Than-Major  Systems  (including  software)  development  involves 
using  Major  Defense  System  Validation  Phase  and  Full-Scale  Development  Phase 
documents  as  models  for  analogous  Less-Than-Major  System  contracts,  or  for 
Less-Than-Major  System  contracts  that  combine  Validation  Phase  & Full-Scale 
Development  Phase  work.  These  models  should  be  tailored  to  the  proposed 
system's  specific  needs  by  eliminating  unnecessary  functions  and  scaling  down 
others.  For  example,  a normal  set  of  specifications.  Test  Plans,  Test 
Procedures,  tests,  and  test  reports  should  probably  still  be  prescribed  in  a 
Less-Than-Major  System  SOW.  However,  the  review  cycles  applicable  to  these 
could  be  simplified  by  reducing  the  size  and  structure  of  the  Government 
comment  coordination  network.  Soliciting  and  reviewing  comments  from  all 
Interested  groups,  but  coordinating  only  the  comments  of  those  legitimately 


DODI  5000.1,  AcQulsition  of  Major  Defense  Systems,  paragraph  II. 
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affected  by  each  document,  could  substantially  reduce  the  coordination  effort 
typically  required.  Again,  elaborate  tasks  in  the  Systems  Engineering, 
Supporting  Project  Management,  Integrated  Logistics  Support,  Human  Factors  and 
Operational/Site  Activation  categories  (and  their  related  data  requirements) 
are  prime  candidates  for  simplification  or  elimination. 


8.0  THE  COMPUTER  PROGRAM  LIFE  CYCLE 


AFR  800-14,  Vol.  II  (paragraph  2-8),  defines  a Computer  Program  Life 
Cycle  distinct  from  the  Major  Defense  System  Acquisition  Life  Cycle,  and 
relates  the  two.  The  Computer  Program  Life  Cycle  consists  of  six  phases. 

These  occur  mainly  in  sequence,  but  overlap  somewhat.  These  phases  are 
termed:  (1)  Analysis,  (2)  Design,  (3)  Coding  and  Checkout,  (4)  Test  and 

Integration,  (5)  Installation,  and  (6)  Operation  and  Support.  AFR  800-14, 

Vol.  II,  also  defines  the  goals  of,  activities  in,  and  milestones  for  each 
phase.  Pei  its  paragraph  2-8,  the  Computer  Program  Life  Cycle  occurs 
separately  for  each  CPCI  developed 

"at  least  once .. .during  the  system  acquisition  life  cycle.  The 
activities  need  not  be  sequential.  Instead,  there  are  potential  loops 
between  all  the  phases." 

8. 1 Nested  Computer  Program  Life  Cycles 

A higher-level  Computer  Program  Life  Cycle  should  also  be  defined  for 
each  strongly  related  set  of  CPCIs,  sometimes  termed  a Software  Subsystem. 

Note  that  MIL-STD-480,  Appendix  E,  defines  "Cl"  to  mean  not  only  an  elementary 
Cl  but  such  a related  set  of  CIs,  including  a Segment.  In  contrast,  this 
guidebook  limits  "Cl"  to  the  lowest  level  aggregate  of  equipment  or  software 
defined  for  Configuration  Management,  and  uses  "Software  Subsystem"  to  mean 
any  group  of  CPCIs  to  which  a separate  Computer  Program  Life  Cycle  applies. 

For  example,  the  Software  Subsystem  of  a large  Command,  Control  and 
Communications  system  might  include  the  Operational  Software  for  each  Segmeh't*^ 
(if  any),  the  Operational  Software  for  the  entire  system,  the  system 
simulation  software,  and  the  T4E  software.  Each  would  comprise  one  or  more 
CPCIs.  The  Computer  Program  Life  Cycle  of  each  CPCI  that  belongs  to  a 
Software  Subsystem  is  nested  in  that  Software  Subsystem's  Computer  Program 
Life  Cycle.  The  terra  Computer  Program  is  used  here  to  mean  either  a CPCI  or  a 
Software  Subsystem. 

8 . 2 Relationship  to  the  Acquisition  Life  Cycle 

Per  AFR  800-14,  Vol.  II  (paragraph  2-8),  a Computer  Program  Life  Cycle 
"may  span  more  than  one  system  acquisition  life  cycle  phase,  or  occur  in  any 
one  phase."  For  example,  high-level  discrete  event  simulation  of  system 
design  alternatives,  to  discern  their  workload  handling  capacities  and  related 
response  times,  should  begin  during  the  Conceptual  Phase  and  should  continue 
with  increasing  refinement  throughout  the  Validation  Phase  and  the  Full-Scale 
Development  Phase.  Similarly,  the  Computer  Program  Life  Cycle  for  the  T&E 
software  might  extend  from  the  Validation  Phase  into  the  Deployment  Phase. 

8.3  Computer  Program  Life  Cycle  Events 

Table  4 summarizes  the  main  types  of  activity  and  product  of  each 
Computer  Program  Life  Cycle  phase.  Table  4 is  based  mainly  on  AFR  800-14, 

Vol.  II,  paragraphs  2-8  and  5-2  through  5-5.  Ambiguities  in  AFR  800-14,  Vol. 
II  have  entailed  some  Interpretation,  however.  E.g.,  note  the  allocation  of 
system-level  DT4E  and  lOT&E  to  the  Installation  Phase.  Also  note  that  some  of 
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the  Computer  Program  Life  Cycle  activi 
relatively  independent  of,  Acquisition 


A.1.  Tradeoff  study  reports. 

2.  Initial  or  Authenticated 
System  Specification  A 
Segment  Specifications 
( if  any) . 


B.l.  Authenticated  Development 

Specification  for  each  CPCl. 

2.  Possible  higher-ievel  speci- 
fication, and  ICD,  changes. 

3.  Parts  of  draft  Product  Speci- 
fications containing  design 
approaches  for  each  CPCI. 


C.  PDR  minutes  and  action  item 
responses . 


A.1.  Functional  flowcharts. 

2.  Detailed  flowcharts. 

3.  Data  format  descriptions. 

4.  Descriptions  of  algorithms 
not  previously  prescribed. 


B.  Preliminary  Product  Specifi- 
cations, including  the  above. 


C. 1.  System,  Segment  (if  any) 

and  CPCI  Test  Plans. 

2.  Preliminary  CP(.  Test  Procedures. 

D.  CDR  minutes  & action  item 
responses . 


Table  U (Continued) 


CODING  AND  CHECKOUT  PHASE 


Activity  Product (s) 


A. 

Coding . 

A-B.  Code. 

b. 

Limited  checkout  of  compiler 
or  assembly  units. 

C. 

Corresponding  logic  4 data 

C.  Altered  Product  Specifications, 

structure  revisions. 

including  compiler/assembly 
listings . 

TEST 

AND  INTEGRATION  PHASE 

Activity 

Product(s) 

A. 

Test  planning. 

A.l.  Final  CPCI  Test  Procedures. 
2.  Segment  (if  any)  and  system- 
level  Test  Procedures. 

B. 

Module  tests. 

B-D. 1.  Test  Reports. 

2.  Computer  Program  coding 
changes . 

C. 

CPCI  tests  (PQT  4 FQT). 

3.  Modified  Product 
Specifications . 

4.  Possible  high-level  specifi- 

D. 

Software  Subsystem  Integration. 

cation,  and  ICD,  changes. 

INSTALLATION  PHASE 
Activity 

A.1.  DT4E  of  any  Segments. 
2.  System-level  DT&E. 


B.  Site  Adaptation  (if  any). 

C.  I0T4E. 


Product (s) 

A. l.  Segment  (if  any)  Test  Reports. 

2.  System-level  DT4E  Test  Reports. 

3.  Computer  Program  coding 
changes . 

Modified  Product  Specifications. 
5.  Possible  higher-level  speci- 
fication, and  ICD  changes. 

B. 1.  Possible  site-specific  coding 

changes.  If  so; 

2.  Version  Description  Documents  4 

3.  Test  Reports. 

C.  10T4E  Test  Reports. 
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Table  <4  (Concluded) 


OPERATION  AND  SUPPORT  PHASE 

Activity 

A.  FOT4E. 


PrQduct(s) 

A.  Analogs  of  Test  and  Integration 
Phase  products. 


b.  Construction,  installation,  & B.  Related  documentation, 
checkout  of  software  maintenance 
4 training  facilities. 


Software  maintenance  4 
modification . 


C.1.  New  software  Versions. 

2.  Version  Description 
Documents . 

3.  Possible  specification 
changes . 

4.  New  or  revised  Test  Plans 
and  Test  Procedures. 

5.  Additional  tests. 

6.  Additional  Test  Reports. 
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APPENDIX  A 


THE  SPECIFICATIONS 


Not  to  be  confused  with  the  Description/Specifications  (see  SOWG,  Section 
C2.2),  the  Specifications  (e.g.,  the  System  Specification)  are  the  RFP 
attachments  that  define  the  system  and  its  parts.  Thus,  the  Specifications 
are  an  essential  part  of  an  RFP  for  a contract  that  includes  software 
development,  since  the  effort  contracted  for  is  best  defined  relative  to 
Specification  provisions.  This  Appendix  summarizes  the  major  Specification 
provisions  affecting  software.  Eventually  the  planned  Software  Acquisition 
lianagement  Guidebook  on  Requirements  Specification  will  be  published  covering 
the  Specifications  in  more  depth. 

An  RFP  may  include  software-related  specifications  of  several  levels  and 
types*,  depending  on  the  contractual  approach,  on  the  Acquisition  Life  Cycle 
Phase  (see  Section  2),  and  on  the  types  of  work  and  products  being  contracted 
for.  Since  the  planned  Software  Acquisition  Management  Guidebook  on 
Requirements  Specification  has  yet  to  be  written,  this  Appendix  is  provided  to 
explain  these  different  kinds  of  specifications  briefly.  Table  A-1  depicts 
the  structure  and  contents  of  the  more  important  types  of  software-related 
specifications . 

The  RFP  for  a Conceptual  Phase  contract  cannot  normally  include  a System _ 
Specification  (discussed  in  Section  A1),  since  an  Initial  System  Specification 
is  a usual  product  of  such  a contract  (see  Section  3* 3)-  However,  the  RFP 
should  Incorporate  any  documents  that  prescribe  system  requirements  or  suggest 
potentially  feasible  designs,  as  direction  to,  or  guidance  for,  the 
contractor.  Such  documents  include  relevant  extracts  of  any  appropriate  ROC, 
plus  specifications  for  analogous  systems,  for  interfacing  systems,  and  for 
any  subsystems  already  defined  that  the  system  being  designed  must 
incorporate . 

In  contrast,  the  RFP  for  a Conceptual  Phase  contract  to  provide  software 
for  feasibility  demonstration  or  system  simulation  should  definitely  include  a 
specification  that  clearly  defines  the  desired  product.  This  could  be  a 
Government-prepared  Computer  Program  Development  Specification  (see  Section 
A3). 

An  RFP  for  Validation  Phase  work  should  include  the  Initial  System 
Specification,  augmented  by  any  other  documents  that  modify  the  system's 
requirements.  In  particular,  the  System  Specifications  should  Include 
specifications  of  interfacing  systems  and  of  any  subsystems  whose  inclusion  in 
the  planned  system  is  required. 


MIL-S-83^90,  Specifications.  Types  and  Forms,  and  MIL-STD-490,  paragraphs 
1.3  & 3*1*3i  briefly  define  the  different  prescribed  specification  types. 
ESD-TR-76-I59  also  discusses  several  types  of  specification. 
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Outlines  of  Software-Related  Specification  Types 


3.2.1  Perforaance  characterlatlca  Function  (first  Function's  name)  CPC  (first  CPC’s  nane) 


Table  A-1  (Continued) 


Table  A-1  (Concluded) 
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490,  Appendix  I.  and  MIL-STI>-483(USAF) , Appendix  III. 

490,  Appendix  VI,  and  HIL-STI>-483(USAF) , paragraph  60.4. 
490,  Appendix  XIII,  and  MIL-STD-483(USAF) , paragraph  60.5 


The  RFP(s)  for  Full-Scale  Development  Phase  contracts  should  each  include 
the  Authenticated  System  Specification  (see  Section  A1),  any  appropriate 
Segment  Specification  (see  Section  A2),  and  a subset  of  the  Allocated  Baseline 
(see  Section  4.3-1)  developed  during  the  Validation  Phase,  by  Government  or 
contractor  personnel.  This  subset  should  comprise  a Computer  Program 
Development  Specification  for  each  CPCI  to  be  developed  under  the  contract 
(see  Section  A3).  Specifications  of  appropriate  type  for  the  CPCIs,  equipment 
CIS,  any  other  Segments,  and  any  other  systems,  with  which  the  software  to  be 
developed  und^r  the  contract  must  interface  should  also  be  provided.  (See 
Sections  A4  and  A5). 

Sof tware-related  Production  Phase  and  Deployment  Phase  RFPs  should  each 
incorporate  the  latest  approved  versions  of  the  System  Specification,  any 
relevant  Si.gment  Specifications,  all  CPCI  Development  and  CPCI  Product 
Specifications  (see  Section  A4),  and  analogous  equipment  specifications  (see 
Section  A5),  pertinent  to  the  planned  software  maintenance  and  modification. 

One  general  policy  is  strongly  recommended:  never  contract  for 
substantial  software  development  without  sufficient,  clear,  specifications. 

For  Operational  Software  and  its  Support  Software  these  should  include  the 
latest  approved  version  of  the  System  Specification,  any  relevant  Segment 
Specifications,  and  Development  Specifications  that  incorporate  a design  of 
validated  feasibility  (see  Section  4.3.2).  Whenever  such  specifications  are 
missing,  incomplete,  internally  inconsistent,  in  conflict  with  other  known 
requirements,  or  inadequately  validated,  software  development  is  premature. 
Before  a software  development  contract  is  let,  further  effort  (perhaps  itself 
contracted  for)  should  rectify  the  deficiencies,  even  if  schedules  thereby 
slip.  As  further  insurance  against  conflict  and  oversight,  these 
specifications'  relative  Order  of  Precedence  should  be  prescribed  in  the 
contract  (see  SOWG,  Section  C2.5.1.)  Failure  to  follow  the  recommended 
procedure  in  past  acquisitions  has  led  to  an  inefficient  software  development 
process  that  sometimes  caused  serious  cost  overruns  and  schedule  slips  in  the 
systems  that  included  this  software.  The  costs  of  sound  specifications  are 
usually  repaid  with  interest  in  problems  avoided  later. 

A 1 . The  System  Specification 

The  System  Specification*  (a  Type  A specification  as  defined  in  MIL-S- 
83490)  is  the  highest  level  specification  of  a system.  A System  Specification 
is  typically  produced  in  at  least  two  versions:  an  Initial  System 

Specification  developed  during  the  Conceptual  Phase  (see  Table  1,  Set  T),  and 
an  Authenticated  System  Specification  (see  Table  2,  Set  E)  developed  during 
the  Validation  Phase.  In  addition,  either  version  of  the  System  Specification 
may  change  as  a result  of  ECP  approvals  after  it  has  been  baselined.  (MIL- 
STD-480  discusses  baselining,  and  control  of  subsequent  specification 
changes . ) 

The  Initial  System  Specification  states  the  overall  system  requirements, 
but  may  identify  the  system's  parts,  and  allocate  requirements  among  them. 


Defined  in  MIL-STD-490,  paragraph  3- 1-3-1;  in  MIL-STD-483( USAF) , 
paragraph  30;  and  in  DI-E-3IOI,  System  Specification. 
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incompletely  or  imperfectly.  These  problems  should  be  resolved  in  the 
Authenticated  System  Specification.  The  Authenticated  (l.e.,  complete  and 
validated)  System  Specification  states  the  functional,  performance,  external 
interface,  design,  and  testing  requirements  of  the  system  as  a whole.  It 
identifies  any  System  Segments  (see  Section  4.3.3);  the  Functional  Areas  (see 
Section  4.3.1)  of  each  Segment  (if  any),  or  otherwise  of  the  system  as  a 
whole;  and  the  equipment  CIs  and  CPCIs  of  each  Functional  Area.  It  allocates 
the  overall  system  requirements  among  Segments  (if  any),  the  Functional  Areas, 
and  the  CIs,  and  it  specifies  any  other  (non-allocated ) requirements  of  each. 
Note  that  the  System  Specification  may  include  constraints  on  the  design  and 
construction  of  the  system  and  its  parts.  For  example,  per  MIL-STD-483(USAF) 
(paragraph  30.5),  System  Specification  paragraph  3-3>8  must  include  software 
design  standards,  identify  prescribed  programming  languages,  and  state  any 
other  software  design  constraints,  for  systems  that  Include  software. 

A2 . The  Segment  Specification 

If  a system  is  tc  be  segmented.  Segment  Specifications  for  one  or  more  of 
its  Segments  are  required  under  some  circumstances.  MIL-STD-483(USAF) , 
paragraph  30.6,  states  the  condit^n»»j,inder  which  Segment  Specifications  are 
mandatory;  i.e., 

"when  a System  or  major  equipment  is  acquired  on  an  incremental  basis  or 

when  a Segment(s)  of  an  existing  System  is  to  undergo  a major 

modif Ication . " 

(Segment  Specifications,  like  System  Specifications,  are  Type  A 
specifications).  Where  optional,  a set  of  Segment  Specifications  may  be 
judged  an  aid  to  specifying  clear  Segment  requirements. 

However,  there  are  good  reasons  to  avoid  Segment  Specifications  if  they 
are  not  mandated.  First,  if  the  System  Specification  properly  characterizes 
and  allocates  requirements  to  each  Functional  area.  Segment  Specifications  may 
be  superfluous.  Each  Segment  comprises  one  or  more  complete  Functional  Areas 
(see  Section  4.3.3).  Thus,  each  Segment's  external  Interface  requirements* 
are  a subset  of  its  Functional  Areas'  interface  requirements.  For  the  same 
reason  each  Segment's  functional,  design,  performance  and  testing  requirements 
are  the  composite  of  the  corresponding  requirements  of  the  Functional  Areas 
that  belong  to  the  Segment.  Thus,  a list  of  each  Segment's  Functional  Areas, 
plus  the  System  Specification's  definition  of  these  Functional  Area 
characteristics  and  requirements,  precisely  define  the  Segment.  Formal 
Segment  definition  can  be  accomplished  by  including  in  the  System 
Specification  a list  of  the  Segments  and  the  Functional  Areas  of  each.  This 
approach  is  recommended. 

Second,  and  most  important,  avoiding  Segment  Specifications  should  reduce 
scattering  of  essential  Information  about  the  system.  Such  scattering  tends 


The  external  interfaces  of  a system,  Segment,  or  Cl  are  its  interfaces 
with  systems.  Segments,  or  CIs  outside  Itself,  in  contrast  to  the 
Interfaces  among  its  parts,  termed  its  internal  interfaces. 
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to  encourage  ignorance  ami  paroct.ial  views  among  the  participants  in  a system 
acquisition,  and  eventually  leads  to  inconsistencies  that  may  entail  extensive 
system  modification  during  System  Integration. 

Third,  having  no  Segment  Specifications  should  save  most  of  the  effort 
and  funds  that  their  development  and  subsequent  updating  would  entail. 

m3-  Computer  Program  Development  Specifications 

As  part  of  the  Allocated  Baseline,  a Computer  Program  Development 
specification*  (Type  b5i)  must  be  produced  for  each  CPCI  to  be  developed.  The 
Computer  Program  Development  Specification  defines  the  requirements  against 
which  the  CPCI  must  be  built.  In  contract,  a Computer  Program  Product 
Specification,  which  must  be  prepared  during  the  Cl's  development,  describes 
ti.c  soltware  as  built.  (See  Section  A4).  The  correspondence  between  each 
CPCl's  Computer  Program  Development  Specification  and  its  Computer  Program 
Product  Specification  is  recognized  by  subtitling  them  "Part  1 of  Two  Parts" 
and  "Part  II  of  Two  Parts",  respectively. 

Tne  Computer  Program  Development  Specification  defines  a CPCl's 
requirements  mainly  in  terms  of  its  functions,  its  performance,  its  interfaces 
with  equipment  and  other  software,  any  constraints  on  its  design,  and  the 
formal  testing  it  must  undergo.  These  statements  of  requirements  are  derived 
from,  and  must  be  consistent  with,  tne  CPCl's  allocated  requirements  as  stated 
in  its  iSegment  Specification  (if  any)  and  in  the  Authenticated  System 
Specification.  MIL-STD-463(USAf ) (paragraph  60.4.3)  permits  incorporating  by 
reference  sucn  System  Specif ication  and  Segment  Specification  requirements  in 
tiie  Computer  Program  Development  Specification.  This  approach  is  recommended, 
mainly  to  reduce  omissions  and  inconsistencies  both  initially  and  during 
updating. 

Tne  Computer  Program  Development  Specification  not  only  references  (or 
restates)  the  System  (and  possible  Segment)  Specification  requirements 
allocated  to  the  CPCI.  It  rusL  also  define  the-SI's  parts,  each  called  a 
Function, ••  and  must  impose  requirements  on  each  Pun^ion,  thus  detailing  the 
system  design  to  at  least  a third  level  (see  Section  4.3.1).  Tluta-,  the 
Functions  ot  a Cl  are  not  merely  a simple  allocation  of  system  furctions.  For 
tne  reasons  explained  in  Section  4.3.2,  the  Computer  Program  Development 
Specification  should  include  all  design  requirements  and  other  assumptions 
used  in  validating  the  system  design,  but  should  omit  unvalidated  design 
detail . 


.'iIL-STC-490,  paragraph  60,  and  HIL-STD-483( USAF) , paragraph  60.4 
prescribe  Computer  Program  Development  Specification  form  4 content.  Dl- 
K-3119A,  Computer  PrQg.>-aro  Development  Specification,  supplements  these 
military  standards  slightly. 

Tne  Functions  of  a Cl  should  not  be  confused  with  functional  requirements 
or  with  the  Functional  Areas  of  a system  (see  Section  4.3. 1). 
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Among  the  many  types  of  requirements  that  a Computer  Program  Development 
Specification  must  define  (explicitly  or  by  reference)  are  the  following: 

a.  each  of  the  CPCI's  Functions,  plus  the  Function's  input,  processing, 
and  output  requirements;  these  should  include  any  algorithms 
encompassed  by  the  verified  design; 

b.  the  CPCI's  external  interfaces,  physical  & functional,  including  the 
characteristics  of  the  computer  on  which  it  operates; 

c.  each  message  type  that  the  CPCI  must  process,  the  message's  format, 
and  its  maximum  data  rate; 

d.  any  program  structure,  programming  standards,  programming  languages, 
or  specific  compilers  prescribed  for  the  CPCI's  development; 

e.  provisions  for  growth  (e.g.,  extra  core,  channel  capacity,  and 
processing  capacity); 

f.  special  requirements  (if  any)  for  handling  classified  data; 

g.  features  (e.g.,  trapping  ints)  to  facilitate  testing; 

h.  any  appropriate  man-machine  interface  requirements  (e.g.,  maximum 
display  densities,  maximum  response  time  to  terminal  operator 
actions) ; 

i.  any  Government-Furnished  (GFP)  software  that  the  CPCI  must 
Incorporate ; 

J.  any  site  adaptation  parameters;  and 

k.  the  CPCI's  overall  workload-handling  capacities. 

besides  these  mandatory  provisions,  a Computer  Program  Development 
Specification  should  specify  the  CPCI's  main  memory  and  auxiliary  storage 
allocations,  plus  other  assumptions.  Included  in  the  verified  system  design 
(see  Section  4.3.2) . 

In  addition,  the  Computer  Program  Development  Specification  must  state 
the  requirements  for  testing  the  CPCI,  but  must  not  specify  detailed  test 
plans  and  test  procedures.  (These  are  normally  prescribed  in  the  SOW  and  CDRL 
as  items  for  contractor  development).  Further,  the  Computer  Program 
Development  Specification  must  relate  each  of  its  testing  requirements  (termed 
Quality  Assurance  (QA)  requirements)  to  one  or  more  of  the  functional, 
performance,  interface,  or  design  requirements.  This  may  be  done  by 
incorporating  a Verification  Matrix  which  identifies  the  QA  requirement(s)  and 
verification  method(s)  applicable  to  each  functional,  performance.  Interface, 
and  design  requirement.  MIL-STD-483(USAF)  (paragraph  60.4.4)  defines  four 
categories  of  Computer  Program  Development  Specification  QA  requirements: 
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a.  For  Computer  Program  Test  & Evaluation  (i.e.,  informal  testing  to 
support  CPCI  development)  QA  requirements  need  be  stated  only  to  the 
extent  necessary  to  collect  relevant  data  unobtainable  later. 

b.  For  PQT,  QA  requirements  need  be  defined  only  to  assure  correct 
operation  of  the  CPCI 's  parts,  if  deemed  necessary  for  simplified 
FQT,  which  tests  the  complete  CPCI.  Otherwise,  QA  requirements  for 
these  two  testing  phases  are  to  be  left  unspecified,  as  contractor 
prerogatives . 

c.  In  contrast,  the  Computer  Program  Development  Specification  mu^t 
spell  out  all  of  the  CPCI 's  FQT  QA  requirements. 

d.  The  Computer  Program  Development  Specification  must  also  spell  out 
CPCI  QA  requirements  that  must  be  deferred  until  Segment-level  (if 
any)  and  system-level  testing. 

A4.  Computer  Program  Product  Specifications 

A CPCI's  Computer  Program  Product  Specification*  (Type  C5)  is  produced 
during  computer  program  development,  to  describe  the  CPCI  ^ built . Usually 
at  least  a preliminary  and  a final  version  are  prepared,  the  latter  describing 
the  complete  and  formally  qualified  CPCI. 

The  Computer  Program  Product  Specification  must  fully  describe  the  CPCI 
as  a whole,  each  of  its  first-level  parts,  termed  Computer  Program  Components 
(CPCs),  and  each  CPC's  structure.  The  CPCs  may  correspond  more  or  less 
exactly  to  the  Functions  defined  in  the  CPCI's  Computer  Program  Development 
Specification  (see  Section  A3),  depending  on  the  Development  Specification's 
design  constraints  and  on  the  developer's  design  approach.  Unless  there  is 
exact  correspondence,  the  contractor  should  be  required  to  include  in  the 
Computer  Program  Product  Specification  a matrix  that  shows  which  CPCs  satisfy 
each  Function. 

The  overall  CPCI  description  must  show  how  the  CPCI's  storage  is 
allocated,  describe  each  data  structure  (i.e.,  file,  table.  Individual  data 
item)  created  or  used,  state  which  CPCs  read  and  which  alter  each  data 
structure  component,  list  any  site  adaptation  data,  incorporate  a top-level 
flowchart  of  the  CPCI,  and  list  & explain  the  impact  of  all  program 
interrupts.  For  each  CPC  a description,  a flowchart,  an  Interface 
description,  a structural  description,  a statement  of  limitations,  and  a 
listing  are  required.  The  CPC's  programming  language  must  also  be  identified. 
(A  preliminary  Computer  Program  Product  Specification  may  omit  listings  of 
CPCs  yet  to  be  coded). 

Per  MIL-STD-i*83( USAF) , paragraph  60.5.**-1,  the  QA  provisions  must 
explicitly  cross-reference  the  Test  Plan  and  Test  Procedures  used  to  qualify 


Defined  in  MIL-STD-^90,  paragraph  130,  and  especially  MIL-STD-<*83  (USAF), 
paragraph  60.5.  DI-E-3120A,  Computer  Program  Product  Specification, 

supplements  these. 
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the  CPCI.  The  QA  provisions  must  specify,  too,  additional  tests  that  assure 
correct  replication  of  the  CPCI.  The  Computer  Program  Product  Specification 
roust  also  state  requirements  for  packaging,  mailing,  shipping  and  storing  the 
storage  media  that  contain  the  CPCI. 

The  use  of  certain  development  methods  may  make  it  desirable  for  the 
Government  to  alter  requirements,  e.g.,  by  DID  modification,  for  the  contents 
of  Computer  Program  Product  Specifications.  For  example,  if  Structured 
Programming  is  used,  conventional  flowcharts  may  be  superfluous. 

A5.  Other  Relevant  Specifications 

Specification  of  equipment,  other  software,  and  other  systems,  with  which 
software  to  be  developed,  maintained  or  modified  must  interface,  is  also 
essential.  MIL-STD-490  and  M1L-STD-483(USAF)  define  specification  types 
applicable  to  equipment  CIs  developed  as  parts  of  Major  Defense  Systems. 
However,  some  of  a CPCI's  interfacing  software,  equipment,  or  other  systems 
may  be  defined  in  commercial  or  other  specifications.  Such  non-standard 
specifications  should  be  reviewed  thoroughly  before  including  them  in  an  RFP, 
to  assure  their  adequacy  (see  Section  4.3.1)  for  their  intended  uses. 


63 


LIST  OF  ABBREVIATIONS 


Abbreviation 

Definition 

AFLC 

Air  Force  Logistics  Command 

AFSC 

Air  Force  Systems  Command 

AFTEC 

Air  Force  Test  and  Evaluation  Center 

ASPR 

Armed  Services  Procurement  Regulations 

ATC 

Air  Training  Command 

BA/PA 

Budget  Authorization/Program  Authorization 

CCB 

Configuration  Control  Board 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

Cl 

Configuration  Item 

CPC 

Computer  Program  Component 

CPCI 

Computer  Program  Configuration  Item 

CPBP 

Computer  Program  Development  Plan 

CRISP 

Computer  Resources  Integrated  Support  Plan 

ChwG 

Computer  Resource  Working  Group 

CWbS 

Contract  Work  Breakdown  Structure 

DCP 

Decision  Coordinating  Paper 

DID 

Data  Item  Description 

DoD 

Department  of  Defense 

DODD 

Department  of  Defense  Directive 

DOD  I 

Department  of  Defense  Instruction 

DSARC 

Defense  Systems  Acquisition  Review  Council 

DT4L 

Development  Test  and  Evaluation 

ECO 

Engineering  Change  Order 

ECP 

Engineering  Change  Proposal 

ESD 

Electronic  Systems  Division 

FCA 

Functional  Configuration  Audit 

FCRC 

Federal  Contract  Research  Center 

F0T4E 

Follow-on  Operational  Test  and  Evaluation 

FQR 

Formal  Qualification  Review 

FQT 

Formal  Qualification  Test 

GFP 

A 

Government-Furnished  Property 

ICD 

Interface  Control  Drawing 

ICWG 

Interface  Control  Working  Group 

IOC 

Initial  Operational  Capability 

I0T4E 

Initial  Operational  Test  and  Evaluation 

OSD 

Office  of  the  Secretary  of  Defense 

0T4E 

Operational  Test  and  Evaluation 

PBS 

Program  Breakdown  Structure 

PCA 

Physical  Configuration  Audit 

PCO 

Procuring  Contracting  Officer 

PDh 

Preliminary  Design  Review 

PM 

Program  Manager 

PMD 

Program  Management  Directive 

PKP 

Program  Management  Plan 

PMRT 

Program  Management  Responsibility  Transfer 

PO 

• 

Program  Office 

LIST  OF  ABBREVIATIONS  (Concluded) 


Definition 


PQT 

QA 

R4D 

RFP 

ROC 

SA 

SAF 

SCN 

SECDEF 

SDR 

SEMP 

30*1 

SOWG 

T&E 

TBD 

TEMP 

TEOA 

V&V 

WBS 


Preliminary  Qualification  Test 
Quality  Assurance 
Research  and  Development 
Request  for  Proposal 
Required  Operational  Capability 
Supplemental  Agreement 
Secretary  of  the  Air  Force 
Specification  Change  Notice 
Secretary  of  Defense 
System  Design  Review 
System  Engineering  Management  Plan 
Statement  of  Work 


Software  Acquisition  Management  Guidebook: 
Statement  of  Work  Preparation 


Test  and  Evaluation 


To  be  Determined 

Test  & Evaluation  Master  Plan 

Test  and  Evaluation  Objectives  Annex  (of  the  PMD) 
Validation  and  Verification 
Work  Breakdown  Structure 
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REFERENCES* 


DEPARTMENT  OF  DEFENSE  PUBLICATIONS 


1 . 

DoD  Directive  5000.26 
21  January  1975 

Defense  Systems  Acquisition  Review 
Council 

2. 

DoD  Instruction  5000.1 
22  December  1975 

Acquisition  of  Major  Defense  Systems 

3. 

DoD  Instruction  5000.2 
21  January  1971 

The  Decision  Coordinating  Paper  (DCP) 
and  the  Defense  Systems  Acquisition 
Review  Council  (DSARC) 

MILITARY  SPECIFICATIONS  AND  STANDARDS 

MIL-S-52779(AD) 
5 April  1974 

Software  Quality  Assurance  Program 
Requirements 

5. 

MIL-S-83490 
30  October  1968 

Specifications,  Types  and  Forms 

6. 

MIL -STD-480 
30  October  1968 

Configuration  Control  - Engineering 
Changes,  Deviations  and  Waivers 

7. 

MIL-STD-483(USAF) 
including  Notice  1 
1 June  1971 

Conf iguj atlon  Management  Practices 
for  Systems,  Equipment,  Munitions, 
and  Computer  Programs 

b. 

MIL-STD-490 
including  Change  2 
15  May  1972 

Specification  Practices 

9. 

MIL-STD-499A(USAl=') 
1 May  19-,  , 

Engineering  Management 

10. 

MIL-STD-881A 
25  April  1975 

Work  Breakdown  Structures  for  Defense 
Materiel  Items 

11. 

M1L-STD-152KUSAF) 
including  Change  2 
2 January  1975 

Technical  Reviews  and  Audits  for 
Systems,  Equipment,  and  Computer 
Programs 

Alh 

FORCE  AND  SUBORDINATE  COMMAND 

DIRECTIVES 

12. 

AF  ASPR  Supplement 
1-2100.50 

Procurement  Plan 

13. 

AFh  57-1 
30  May  1975 

Required  Operational  Capabilities 
( ROCs ) 
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u. 

AFR  70-15 
16  April  1976 

Source  Selection  Policy  and  Procedures 

15. 

AFR  80-14 
10  February  1975 
AFSC  Sup.  1 
16  June  1975 

Test  and  Evaluation 

16. 

AFR  800-2 
including  Change  1 

30  April  1975 
AFSC  Sup.  1 

18  October  1974 
ESD  Sup . ’ 

31  July  1975 

Program  Management 

17. 

AFR  800-3 
including  Change  1 
25  February  1975 

Engineering  for  Defense  Systems 

18. 

AFR  800-14,  Vol.  II 
26  September  1975 

Acquisition  and  Support  Procedures 
for  Computer  Resources  in  Systems 

19. 

AFSCP  800-3 
9 April  1976 

A Guide  for  Program  Management 

20. 

AFSCR  70-9 
16  August  1974 
ESD  Sup . 1 
20  October  1975 

Source  Selection  Procedures 

21 . 

AFSCR  80-15 
31  December  1974 

R&D  Source  Selection  Policy  and 
Guidance 

DATA 

ITEM  DESCRIPTIONS** 

22. 

DI-E-3101 

System  Specification 

23. 

DI-E-3119A 

Computer  Program  Development 
Specification 

24. 

DI-E-3120A 

Computer  Program  Product 
Specification 
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REFERENCES  (Concluded) 


OTHER 


25.  Joseph  T.  Connolly,  Software  Acquisition  Management  Guidebook: 
Re&ulatlons.  Specifications  and  Standards.  ESD-TR-75-91  (MTR-3080, 
Contract  F19628-75-C-0001 , The  MITRE  Corporation,  Bedford,  Mass.)> 
October  1975. 

26.  S.  R.  Hagan  and  C.  W.  Knight,  An  Air  Force  Guide  for  Monitoring  and 
Reporting  Software  Development  Status.  ESD-TR-75-85  (MTR-3051,  Contract 
F19628-75-C-0001 , The  MITRE  Corporation,  Bedford,  Mass.),  September  1975. 

27.  N.  E.  Bolen,  An  Air  Force  Guide  to  Contracting  for  Software  Acquisition. 
ESD-TR-75-365,  (MTR-3118,  Contract  F19628-76-C-0001 , The  MITRE 
Corporation,  Bedford,  Mass.),  January  1976. 

28.  W.  L.  Schoeffel,  An  Air  Force  Guide  to  Software  Documentation 
Requirements.  ESD-TR-76-159  (MTR-3180,  Contract  F19628-76-C-000 1 , The 
MITRE  Corporation,  Bedford,  Mass.),  June  1976. 

29.  D.  R.  Peterson,  Software  Acquisition  Management  Guidebook;  Software 
Itave^opnignt  and  Maintenance  Facilities.  (MTR-3330,  Contract  F19628-77-C- 
0001,  The  MITRE  Corporation,  Bedford,  Mass.),  to  be  published. 

30.  J.  B.  Glore  and  VI.  L.  Bjerstedt,  Software  Acquisition  Management 
flyMebook;  statement  of  Work  Preparation.  ESD-TR-77-16  (MTR-31911, 

Contract  F19628-77-C-0001 , The  MITRE  Corporation,  Bedford,  Mass.), 

January  1977. 


The  Regulations,  Specifications,  Standards,  and  DIDs  cited  are  those  in 
effect  at  the  time  the  research  for  the  guidebook  was  completed.  Since 
that  time  new  versions  of,  or  changes  to,  some  of  them  have  been  issued. 
Readers  who  want  the  latest  version  of  a reference  should  check  official 
sources . 

Additional  DIDs  are  referenced  in  Tables  1-3. 
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